I recently spent a day wiring up OpenLineage events into AWS DataZone, seeding static lineage for Firehose streams and Glue crawlers, debugging fileb:// encoding issues, fighting DataZone’s dual authorization model, and provisioning Lakehouse environments — all to get a visual graph in a portal showing that my S3 bucket feeds a Glue table.
When I finally got the lineage tab working, I looked at the graph and realised: this tells me nothing about my users.
The graph showed pweb-prod-events → RawFirehoseDelivery → table/pweb_raw_prod/all_events. Technically correct. Completely useless for the question I actually cared about: are users who sign up for Noteble actually generating their first document?
That question was already answered by a 30-line SQL view I’d written hours earlier.
Two Different Questions, Two Different Tools
The confusion starts because “data lineage” and “business flow analytics” sound like they should be the same thing. They’re not.
Data lineage answers: Where did this data come from? What transformed it? Who consumed it?
Business flow analytics answers: What are my users doing? Where do they drop off? Who’s about to churn?
| Data Lineage | Business Flow Analytics | |
|---|---|---|
| Question | Where did this Parquet file come from? | What do users do after signing up? |
| Audience | Data engineers, compliance | Product managers, founders |
| Visualisation | DAG of datasets and transforms | Funnels, cohort heatmaps, user tables |
| Tools | DataZone, OpenLineage, Apache Atlas, DataHub | Athena views, dashboards, Mixpanel, Amplitude |
| Value at 10 users | Almost none | Immediate |
| Value at 10,000 users | Moderate (audit, debugging) | Critical |
| Value at 1M users | High (compliance, impact analysis) | Critical |
The enterprise data world has spent two decades building sophisticated lineage tools because at scale, with hundreds of data pipelines across dozens of teams, knowing where data came from is genuinely hard. When a compliance audit asks “show me every system that touches PII,” lineage is the answer.
But if you’re a small team shipping a SaaS product, that’s not your problem. Your problem is: are people using the thing I built?
The Two Types of Data Product
“Data product” is one of the most overloaded terms in the industry. When someone says “we built a data product,” they could mean two fundamentally different things:
Infrastructure Data Product
This is the pipeline: events flowing into EventBridge, Firehose delivering Parquet files to S3, Glue crawlers cataloging schemas, Athena workgroups with cost controls. It’s the plumbing that makes data queryable. Data engineers build it, and when it works well, nobody thinks about it.
An infrastructure data product answers: can I query this data? It says nothing about what the data means or what to do with it.
Business Data Product
This is what comes out of the pipeline: a funnel showing that 60% of signups never generate their first document, a cohort heatmap showing activation rates improving week over week, a churn table flagging three users who went silent this week, a security dashboard showing spam attacks targeting the contact form.
A business data product answers: what should I do next?
How They Relate
The infrastructure data product is the foundation. The business data product is the value. You can’t have the second without the first, but building only the first is like laying pipes to a house with no taps.
| Layer | What it is | Who cares | Example |
|---|---|---|---|
| Infrastructure data product | The pipeline and storage | Data engineers | EventBridge → Firehose → S3 → Glue → Athena |
| Business data product | The answers | Founders, PMs | “38% of this week’s cohort activated within 7 days” |
| Delivery surface | How you see the answers | Everyone | Admin dashboard, QuickSight, Slack alert |
Most “data product” content online — data mesh, data contracts, data governance, schema registries — focuses on the infrastructure layer. That’s because enterprise teams have genuine complexity there: hundreds of pipelines, dozens of teams, compliance mandates.
But for a small team, the infrastructure should be invisible. You set it up once (a day’s work with SAM, EventBridge, and Firehose), and then you spend your time on the business data product — the views, the dashboard, the alerts that tell you whether your product is working.
The mistake I made: I spent a full day on DataZone lineage (infrastructure visibility) before building the six SQL views that actually answer my product questions. The lineage graph is technically correct and completely useless for daily decisions. The dashboard with funnel, cohorts, and churn is what I open every morning.
What Enterprise Data Governance Looks Like
Enterprise setups invest heavily in data governance platforms. A typical stack:
Data Sources → ETL/ELT Pipelines → Data Warehouse → BI Tools
↓ ↓ ↓ ↓
Schema Transform Curated Dashboard
Registry Lineage Datasets Lineage
↓ ↓ ↓ ↓
Data Governance Platform
(Catalog + Lineage + Glossary + Access Control)Tools in the enterprise stack:
- Apache Atlas or DataHub — open-source metadata platforms (require Kubernetes, Kafka, Elasticsearch)
- Collibra or Alation — commercial data intelligence platforms (£100k+/year)
- AWS DataZone or Google Dataplex — cloud-native governance
- OpenLineage — CNCF standard for lineage event capture
- Great Expectations or Soda — data quality monitoring
- dbt — transformation layer with built-in lineage
An enterprise team might have 50 data pipelines across 10 teams, each reading from different source systems. When someone changes a column in the CRM, lineage tells them which 15 downstream dashboards will break.
This makes sense at scale. When you have hundreds of datasets, dozens of producers, and compliance requirements that mandate audit trails — lineage is table stakes.
What a Small Team Actually Needs
A small team (1–15 engineers, one product, pre-PMF or early growth) has a fundamentally different problem. You don’t have 50 data pipelines. You have one product and you need to know if people are using it.
Your real questions:
- Funnel: How many signups convert to active users? Where do they drop off?
- Cohorts: Are this week’s signups activating faster than last week’s?
- Churn: Which users stopped using the product? When?
- Feature adoption: Which features are people actually using?
- Revenue: How fast are free users converting to paid?
- Security: Is someone spamming my contact form?
None of these require a governance platform. They require well-defined events and SQL views.
The Small Team Stack
Lambda → emit_event() → EventBridge → Firehose → S3 (Parquet)
↓
Enrichment Lambda (joins user context)
↓
Firehose → S3 enriched/ (Parquet)
↓
Glue Catalog → Athena → DashboardTotal monthly cost: Under £20.
What you get:
- Every business event captured permanently in S3
- User context (email, plan, name) joined at write time
- SQL views for funnels, cohorts, churn, and feature engagement
- An admin dashboard that shows you what’s happening right now
What you don’t need yet:
- A metadata catalog portal
- OpenLineage integration
- Schema registry
- Data quality monitoring
- Column-level lineage
- Subscription-based access control for datasets
The Views That Actually Matter
Here are the six SQL views that answer 90% of a small team’s product questions. All run against an enriched event stream where user context is already joined.
1. User Journey Funnel
How many users reach each milestone?
SELECT
lifecycle_stage,
COUNT(*) AS users
FROM (
SELECT user_id,
CASE
WHEN first_paid_at IS NOT NULL THEN 'converted'
WHEN first_share_at IS NOT NULL THEN 'engaged'
WHEN first_doc_at IS NOT NULL THEN 'activated'
WHEN first_session_at IS NOT NULL THEN 'tried'
ELSE 'signed_up_only'
END AS lifecycle_stage
FROM user_milestones
) GROUP BY lifecycle_stage;This one view tells you more about your product’s health than any lineage graph ever will.
2. Activation Cohorts
Are we getting better at activating users?
Weekly signup cohorts with the percentage that reached each milestone within 7 days:
| Cohort Week | Size | Session 7d | Doc 7d | Paid 30d |
|---|---|---|---|---|
| Jun 02 | 45 | 62% | 38% | 8% |
| Jun 09 | 52 | 71% | 44% | 11% |
| Jun 16 | 48 | 68% | 42% | 9% |
If the percentages are trending up, your product changes are working. If they’re flat or declining, no amount of data governance will help.
3. Churn Signals
Who’s about to leave?
SELECT email, plan, days_since_last_activity,
CASE
WHEN days_inactive >= 30 THEN 'dormant'
WHEN days_inactive >= 14 THEN 'at_risk'
WHEN days_inactive >= 7 THEN 'cooling'
ELSE 'active'
END AS risk_level
FROM user_activity
WHERE total_events >= 3
ORDER BY days_since_last_activity DESC;This is actionable. You can email at-risk users. You can call them. You can’t do anything useful with a lineage graph showing dataset provenance.
4. Session-to-Document Conversion
Is the core product loop working?
SELECT day,
sessions_started,
docs_generated,
ROUND(100.0 * docs_generated / NULLIF(sessions_started, 0), 1) AS conversion_pct
FROM session_doc_daily;5. Feature Engagement
What are people actually using?
Sum across all users: recording sessions, AI-generated docs, AI chat messages, shares, architecture docs. Sort by count — the most-used feature is at the top. No lineage required.
6. Security Events
Is someone attacking my product?
SELECT product, reason, field, COUNT(*) AS blocked
FROM security_events
WHERE detail_type LIKE '%.spam_blocked%'
GROUP BY product, reason, field
ORDER BY blocked DESC;When you see product=systemdox, reason=gibberish, field=name in the results, you know exactly what’s happening and where.
When to Add Data Governance
Data governance becomes valuable when:
- Multiple teams produce and consume data independently. Lineage helps team B understand what team A’s pipeline does without asking.
- Compliance requires audit trails. GDPR, SOX, or industry regulations mandate knowing where PII flows.
- Schema changes break downstream consumers. Impact analysis (who’s affected by this column rename?) requires lineage.
- New engineers can’t find the data they need. A searchable catalog with business glossary speeds up onboarding.
- Data quality issues are hard to diagnose. Column-level lineage traces a bad value back to its source.
For a small team, the right order is:
- Events first — emit well-defined business events from your application
- Views second — write SQL that answers your product questions
- Dashboard third — build a UI that shows the views in one place
- Governance later — add DataZone, lineage, glossary when you have multiple data consumers
Enterprise vs Small Team: Side by Side
| Dimension | Enterprise | Small Team |
|---|---|---|
| Primary question | “Where does this data come from?” | “Are users using the product?” |
| Data sources | 50+ databases, APIs, SaaS tools | 1 product, 1 database, 1 event bus |
| Teams consuming data | 10+ (analytics, ML, compliance, ops) | 1 (the founding team) |
| Governance tools | DataZone/Collibra + OpenLineage + Schema Registry + Data Quality | Optional — add when needed |
| Lineage value | High (impact analysis, compliance) | Low (you know where the data comes from — you wrote the code) |
| Business analytics value | High | Critical |
| Monthly data platform cost | £10k–100k+ | Under £20 |
| Time to first insight | Weeks (setup, catalog, permissions) | Hours (SQL view + dashboard) |
| What breaks without it | Compliance audits, cross-team coordination | Nothing — you already know your own code |
The Architecture That Scales Both Ways
The good news: you can start with the small team stack and evolve to enterprise governance without re-architecture. The event stream is the same either way.
graph TD
subgraph "Always needed"
A[Business Events] --> B[EventBridge]
B --> C[S3 Data Lake]
C --> D[Athena Views]
D --> E[Admin Dashboard]
end
subgraph "Add when you need it"
C --> F[Glue Catalog]
F --> G[DataZone Portal]
G --> H[Lineage Graphs]
G --> I[Business Glossary]
G --> J[Access Control]
end
style A fill:#2563eb,color:#fff
style E fill:#2563eb,color:#fff
style G fill:#7c3aed,color:#fffThe left side (events → S3 → Athena → dashboard) costs under £20/month and answers your business questions immediately. The right side (DataZone, lineage, glossary) costs almost nothing to add later — DataZone is free, lineage events are just API calls — but provides diminishing returns until you actually have multiple data consumers.
Key Takeaways
-
Data lineage and business flow analytics solve different problems. Lineage tells you where data came from. Analytics tells you what users are doing. Small teams need the latter first.
-
Don’t start with governance. Start with events, then views, then a dashboard. Add DataZone and lineage when you have compliance requirements or multiple teams consuming data independently.
-
Six SQL views answer 90% of product questions. User journey funnel, activation cohorts, churn signals, session conversion, feature engagement, and security events. Write these before investing in any governance platform.
-
The same event stream powers both. CloudEvents on EventBridge → S3 → Athena is the foundation for both the simple dashboard and the enterprise governance portal. You don’t need to re-architect when you grow.
-
Enterprise tools solve enterprise problems. If you have 50 data pipelines and 10 consumer teams, invest in lineage and cataloging. If you have one product and five engineers, invest in understanding your users.
The most valuable data tool for a small team isn’t a lineage graph — it’s a SQL view that shows whether people are using your product.
Need help building a data analytics stack that scales with your team? See our Architecture & Cloud Engineering services or get in touch for a free architecture review.