Lambda Lessons from Production: The Bugs Nobody Warns You About

I spent last weekend shipping a new feature to production. A chatbot that lets business owners update their website content through conversation — type “update Monday hours to 9am-6pm” and the AI parses it, validates it, writes it to DynamoDB, and triggers a site rebuild. Straightforward Lambda-behind-API-Gateway stuff.

It took four deploys to get it working. Not because the code was wrong, but because Lambda has a collection of behaviours that are perfectly logical once you understand them and completely invisible until you don’t.

These aren’t hypothetical pitfalls from a conference talk. Every one of them cost me actual debugging time in the past week. If you’re building on Lambda — especially if you’re coming from containers or traditional servers — this might save you a few hours.

The Layer That Wasn’t There

Lambda Layers sound simple. You package shared code into a zip, attach it to your function, and import from it. The documentation even shows you how.

What the documentation doesn’t emphasise enough is this: Python Lambda layers must place their modules under a python/ directory inside the zip. Not at the root. Under python/.

Here’s what I had:

layers/common/
  common/
    __init__.py
    content_types.py
    auth.py

This deploys to /opt/common/ at runtime. Looks reasonable. Except Python’s sys.path in Lambda includes /opt/python, not /opt. So from common.content_types import validate fails with No module named 'common'.

The fix:

layers/common/
  python/
    common/
      __init__.py
      content_types.py
      auth.py

Now it deploys to /opt/python/common/ and the import works.

flowchart LR
    subgraph WRONG["Wrong: modules at /opt/"]
        direction TB
        OPT1["/opt/"]
        COM1["common/"]
        PY1["content_types.py"]
        OPT1 --> COM1 --> PY1
    end

    subgraph RIGHT["Correct: modules at /opt/python/"]
        direction TB
        OPT2["/opt/"]
        PYD["python/"]
        COM2["common/"]
        PY2["content_types.py"]
        OPT2 --> PYD --> COM2 --> PY2
    end

    subgraph PATH["Python sys.path includes"]
        P1["/opt/python"]
        P2["/var/task"]
        P3["/var/runtime"]
    end

    RIGHT -. "importable" .-> PATH
    WRONG -. "NOT on sys.path" .-> PATH

    style WRONG fill:#fde2e2,stroke:#c0392b
    style RIGHT fill:#e2f4e6,stroke:#27ae60
    style PATH fill:#f0f4ff,stroke:#4f46e5

The maddening part? I had five other functions using the same layer — and they’d been working for weeks. The layer was broken the entire time. Those functions just hadn’t been invoked since the last deploy that happened to have a cached build artefact with the correct structure. The moment a fresh build went through, every function using that layer would have failed.

Lesson: Lambda layers for Python go in python/. For Node, it’s nodejs/node_modules/. For Java, it’s java/lib/. The runtime-specific prefix is not optional.

The Silent 401

This one is subtle enough that I’d call it a design flaw in API Gateway.

When you attach a Lambda REQUEST authoriser to your API Gateway, you configure an Identity section that tells API Gateway which headers to look at:

Auth:
  Authorizers:
    MyAuthoriser:
      Identity:
        Headers:
          - Authorization
          - X-API-Key
          - X-Tenant-Id
        ReauthorizeEvery: 300

The documentation frames this as “identity sources for caching.” What it doesn’t spell out is the behaviour when ReauthorizeEvery is greater than zero: API Gateway requires ALL listed headers to be present in the request. If any are missing, it returns 401 immediately — without ever calling your authoriser Lambda.

No log entry. No CloudWatch invocation. No error message beyond {"message": "Unauthorized"}. Your authoriser code is perfect. Your token is valid. But the request is rejected at the gateway level because a header you listed as an identity source wasn’t sent.

In my case, the portal sends Authorization but not X-API-Key or X-Tenant-Id (those are for API-key-based auth, not browser sessions). The authoriser Lambda handles both paths gracefully. But API Gateway never gave it the chance.

The fix was either:

• Remove optional headers from the identity source
• Set ReauthorizeEvery: 0 (disable caching, always invoke the Lambda)

I went with both. The authoriser runs in under 50ms — caching saves nothing meaningful, and the “all headers required” behaviour is a trap waiting to fire again.

sequenceDiagram
    participant Browser
    participant APIGW as API Gateway
    participant Auth as Authoriser Lambda

    Note over Browser,Auth: Scenario A: All identity headers present
    Browser->>APIGW: POST /api (Auth + X-API-Key + X-Tenant-Id)
    APIGW->>Auth: Invoke authoriser
    Auth-->>APIGW: Allow (policy + context)
    APIGW-->>Browser: 200 OK

    Note over Browser,Auth: Scenario B: Optional header missing
    Browser->>APIGW: POST /api (Auth only)
    APIGW--xBrowser: 401 Unauthorized (Lambda never called)
    Note right of APIGW: No CloudWatch logs.<br/>No error detail.<br/>Just silence.

Lesson: Only list headers in your authoriser’s Identity section that every client will always send. If you have multiple auth paths (JWT + API key), keep only the common denominator in the identity source.

The Cold Start That Wasn’t a Cold Start

Everyone knows about Lambda cold starts. Function hasn’t been called in a while, the container is recycled, the next invocation pays an init penalty. Fair enough.

What caught me was a different flavour: the deployment cold start. When you deploy a new version of a function, Lambda doesn’t update the existing containers. It provisions new ones. Every function gets a fresh cold start after deploy, regardless of how recently it was invoked.

This matters when your function imports heavy libraries at module scope. If you’ve got import boto3 at the top level (you should — it’s the right pattern for reuse across invocations), that’s fine. But if you’re also importing, say, a ULID library that pulls in typing_extensions, and that dependency isn’t in the Lambda runtime…

[ERROR] Runtime.ImportModuleError: Unable to import module 'handler': 
No module named 'typing_extensions'

This only surfaces on cold start. If you tested locally with pip install python-ulid in a venv, everything works. If you tested with SAM local invoke, it works (SAM uses Docker with a full Python environment). It fails only in production, only on cold start, and the error message points at a dependency you never explicitly imported.

Lesson: Know exactly what’s in the Lambda runtime and what isn’t. Python 3.11 on Lambda includes boto3, botocore, urllib3, and the standard library. Everything else — even common packages like typing_extensions — needs to be bundled in your deployment package or a layer. When in doubt, use stdlib alternatives: secrets.token_hex() instead of third-party ID generators.

IAM: The Prefix Trap

SAM generates IAM role names from your stack name and logical resource ID. A function called AdminContentChatFunction in a stack called puglieseweb-data-prod gets a role like puglieseweb-data-prod-AdminContentChatFunctionRole-a9B3xK2.

If your deploy role has IAM permissions scoped to a prefix — say, arn:aws:iam::*:role/puglieseweb-data-* — this works. But when you add a second SAM stack in the same repo with a different prefix (puglieseweb-conversation-admin-*), the deploy fails:

CREATE_FAILED  AWS::IAM::Role  ListConversationsFunctionRole
Encountered a permissions error performing a tagging operation

“Permissions error performing a tagging operation.” Not “access denied on CreateRole.” Not “insufficient permissions.” A tagging error. Because CloudFormation tries to tag the role during creation, and tagging requires the same resource scope as creation.

The fix: widen the IAM prefix to match all stacks the deploy role manages.

Lesson: When you add a new SAM stack to a repo, update the deploy role’s IAM policy to include the new stack name prefix. Check Lambda, IAM, and DynamoDB resource scopes. The error messages will not tell you what’s actually wrong.

CORS: The Error That Hides Other Errors

API Gateway has a Cors configuration that handles preflight OPTIONS requests. You set it up, test it, it works. Then your authoriser rejects a request and the browser shows:

Access to fetch has been blocked by CORS policy: 
No 'Access-Control-Allow-Origin' header is present

The actual error was 401 Unauthorized. But the browser never shows you that because the 401 response doesn’t have CORS headers, so the browser blocks it before your JavaScript can read the status code.

API Gateway’s Cors configuration only applies to successful responses and OPTIONS preflight. For error responses generated by API Gateway itself (authoriser failures, throttling, 5xx), you need GatewayResponses:

GatewayResponses:
  DEFAULT_4XX:
    ResponseParameters:
      Headers:
        Access-Control-Allow-Origin: "'*'"
        Access-Control-Allow-Headers: "'Content-Type,Authorization'"
        Access-Control-Allow-Methods: "'GET,POST,OPTIONS'"
  DEFAULT_5XX:
    ResponseParameters:
      Headers:
        Access-Control-Allow-Origin: "'*'"

Without this, every error from your API looks like a CORS error to the browser. You’ll waste time debugging CORS when the real issue is authentication, permissions, or a crashed function.

flowchart TD
    REQ["Browser sends POST /api"] --> APIGW["API Gateway"]
    APIGW --> AUTH{"Authoriser<br/>result?"}

    AUTH -->|"Allow"| FN["Lambda Function"]
    FN --> OK["200 + CORS headers"]
    OK --> BROWSER_OK["Browser reads response"]

    AUTH -->|"Deny"| ERR{"GatewayResponses<br/>configured?"}
    ERR -->|"Yes"| ERR_CORS["401 + CORS headers"]
    ERR_CORS --> BROWSER_ERR["Browser shows: 401 Unauthorized"]

    ERR -->|"No"| ERR_NO_CORS["401, no CORS headers"]
    ERR_NO_CORS --> BROWSER_BLOCK["Browser shows: CORS error<br/>(real error hidden)"]

    style BROWSER_OK fill:#e2f4e6,stroke:#27ae60
    style BROWSER_ERR fill:#fff3d6,stroke:#e08e0b
    style BROWSER_BLOCK fill:#fde2e2,stroke:#c0392b

Lesson: Always add GatewayResponses for DEFAULT_4XX, DEFAULT_5XX, UNAUTHORIZED, and ACCESS_DENIED with the same CORS headers as your main configuration. Do this in every SAM template that has an API Gateway. No exceptions.

The Rollback That Wasn’t

CloudFormation tries to be helpful when a deploy fails: it rolls back to the previous state. Usually this works. But if the rollback itself fails (maybe you deleted a resource that CloudFormation expected to still exist, or an IAM permission was removed between deploys), you get:

Stack is in ROLLBACK_FAILED state and can not be updated

Your stack is now stuck. You can’t update it. You can’t roll it back further. The only option is to delete the stack entirely and recreate it. If the stack owns DynamoDB tables with data, you’d better have DeletionPolicy: Retain on them, or that data is gone.

Lesson: Set DeletionPolicy: Retain and UpdateReplacePolicy: Retain on every stateful resource (DynamoDB tables, S3 buckets, KMS keys). It costs nothing and saves everything when a deploy goes wrong.

What I’d Actually Do Differently

If I were starting a new Lambda project tomorrow, knowing what I know after this week:

• Layer structure: Create a python/ directory from day one. Not after the first mysterious import failure.
• Authoriser identity: Only Authorization in the identity source. Add API-key and tenant headers to the authoriser Lambda’s logic, not to the gateway’s identity config.
• GatewayResponses: Copy them from your last working template into every new one. Make it muscle memory.
• Deploy role prefixes: Use {project}-* not {project}-{component}-*. You’ll add components. The prefix should be stable.
• No third-party deps for trivial things: secrets.token_hex(16) is a perfectly good unique ID. You don’t need a ULID library that brings in typing_extensions which isn’t in the Lambda runtime.
• Fitness tests: We now have automated checks that flag missing GatewayResponses and misconfigured authoriser identity sources. The bugs I hit this week won’t happen again — for any service in the organisation. That’s the real win.

None of these are complicated. Most of them are one-line fixes. The hard part is knowing they exist before they bite you at midnight on a weekend deploy.

Lambda is a genuinely good platform. The execution model is sound, the pricing is fair, and the integration with the rest of AWS is deep. But it has sharp edges that the documentation glosses over, and they tend to surface at the worst possible time.

Every production incident is a future fitness test. That’s the only way to make sure the same class of bug never recurs across an entire organisation.