• Home
  • >
  • DevOps News
  • >
  • Debugging Tests with Distributed Tracing for LocalStack Apps – InApps 2025

Debugging Tests with Distributed Tracing for LocalStack Apps – InApps is an article under the topic Devops Many of you are most interested in today !! Today, let’s InApps.net learn Debugging Tests with Distributed Tracing for LocalStack Apps – InApps in today’s post !

Key Summary

  • Overview: The article explores how to use distributed tracing to debug tests for applications running on LocalStack, a local AWS cloud emulator, as presented by InApps Technology in 2022.

  • Key Points:

    • LocalStack Overview:
      • An open-source tool that emulates AWS services (e.g., S3, Lambda, DynamoDB) locally for development and testing.
      • Enables developers to build and test cloud-native apps without incurring AWS costs or needing internet access.
    • Distributed Tracing:
      • A debugging technique that tracks requests across microservices or components, providing visibility into performance and errors.
      • Captures end-to-end request flows, including latency, dependencies, and failure points.
    • Why Use Distributed Tracing with LocalStack?:
      • LocalStack apps often mimic complex AWS architectures (e.g., Lambda + SQS + API Gateway).
      • Tracing helps identify issues in local environments, such as misconfigured services, timeouts, or data flow errors.
      • Simplifies debugging tests by pinpointing failures in distributed systems.
    • Key Tools for Distributed Tracing:
      • OpenTelemetry: Open-source framework for collecting traces, metrics, and logs; integrates with LocalStack.
      • Jaeger: Distributed tracing system for visualizing request flows, compatible with OpenTelemetry.
      • Zipkin: Alternative tracing tool with similar functionality.
      • AWS X-Ray: Native AWS tracing service, partially supported by LocalStack for local testing.
    • Implementation Steps:
      • Set Up LocalStack:
        • Install LocalStack via Docker: docker run -p 4566:4566 localstack/localstack.
        • Configure services (e.g., Lambda, S3) using LocalStack’s CLI or SDKs.
      • Instrument Application:
        • Add OpenTelemetry SDK to your app (e.g., Python, Node.js) to generate traces.

Example (Python):
from opentelemetry import trace

from opentelemetry.exporter.jaeger import JaegerExporter

from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())

jaeger_exporter = JaegerExporter(agent_host_name=”localhost”, agent_port=6831)

trace.get_tracer_provider().add_span_processor(jaeger_exporter)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span(“my-operation”):

    # Your code here

  • Configure Tracing Backend:
    • Run Jaeger locally: docker run -p 6831:6831 -p 16686:16686 jaegertracing/all-in-one.
    • Access Jaeger UI at http://localhost:16686 to view traces.
  • Run Tests:
    • Execute tests against LocalStack (e.g., using boto3 for AWS SDK in Python).
    • Trigger requests to generate traces (e.g., API calls, Lambda invocations).
  • Analyze Traces:
    • Use Jaeger or Zipkin to visualize request paths, latencies, and errors.
    • Identify bottlenecks (e.g., slow DynamoDB queries) or misconfigurations (e.g., missing IAM roles).
  • Common Debugging Scenarios:
    • Lambda timeouts due to incorrect LocalStack endpoint configuration.
    • SQS message failures caused by queue misrouting.
    • API Gateway errors from malformed request mappings.
  • Best Practices:
    • Instrument all services (e.g., Lambda, API Gateway) for complete trace coverage.
    • Use unique trace IDs to correlate requests across components.
    • Minimize tracing overhead by sampling only critical requests in production-like tests.
    • Integrate tracing with CI/CD pipelines for automated test debugging.
  • Use Cases:

    • Debugging microservices apps locally before deploying to AWS.
    • Testing serverless architectures (e.g., Lambda + SNS) in CI/CD environments.
    • Validating integrations between AWS services in a cost-free local setup.
  • Benefits:

    • Enhances visibility into complex, distributed LocalStack apps.
    • Speeds up debugging by isolating failures in specific components.
    • Reduces reliance on AWS cloud for testing, saving costs.
    • Improves test reliability with end-to-end request tracking.
  • Challenges:

    • Initial setup of tracing tools (e.g., OpenTelemetry, Jaeger) requires configuration.
    • LocalStack’s emulation may not fully replicate AWS behavior, causing trace discrepancies.
    • Overhead of instrumentation can slow down local tests if not optimized.
    • Limited community support for advanced tracing in LocalStack compared to AWS X-Ray.
  • Conclusion: In 2022, using distributed tracing with tools like OpenTelemetry and Jaeger, as outlined by InApps Technology, significantly improves debugging of tests for LocalStack-based applications, offering clear insights into distributed systems, though setup complexity and LocalStack limitations require careful management.

Read more about Debugging Tests with Distributed Tracing for LocalStack Apps – InApps at Wikipedia

You can find content about Debugging Tests with Distributed Tracing for LocalStack Apps – InApps from the Wikipedia website

Oguzhan Ozdemir

Oguzhan is a solutions engineer at Thundra.

LocalStack has an amazing community, and everybody loves the product. Not only does it save organizations money, but it also enables a highly efficient local development and test loop for developing cloud apps.

Since the local testing framework is provided by LocalStack, the need to set up multiple testing environments on the cloud disappears and distributed microservice applications can easily be developed, tested and deployed with green tests.

Not all the time, though! It’s a fact that tests fail. Some erroneous tests make developers tear their hair out looking for the root cause in log piles.

This is when Thundra comes to the rescue with Foresight for those who use LocalStack on their continuous integration (CI) pipeline. Foresight helps developers understand the root causes of failing and long-running tests easily and quickly. It gives rich insights about the passed, failed and aborted tests of the test suites.

Sample App

We’ve developed a small application that’ll spawn multiple AWS services on your local machine with LocalStack and monitor your distributed architecture and its tests with Thundra Foresight.

An AWS Lambda function, let’s call this Lambda #1, with an HTTP endpoint that takes a request, then writes to an Simple Queue Service (SQS) queue for further processing. Then, Lambda #1 writes the same request into a DynamoDB database for bookkeeping purposes. The SQS queue passes the incoming message to the processor Lambda, our Lambda #2.

Once the processing is successful, Lambda #2 notifies a Simple Notification Service (SNS) topic, which triggers our archive Lambda, Lambda #3. When Lambda #3 is triggered, it will update the request’s record on DynamoDB. It will also write the result of this whole request into a file and put it in an S3 bucket.

This might seem a lot, but all it does is pass the body of an HTTP request between different services and manipulate it along the way. This project is developed by Thundra’s software engineer Tolga Takır and the source code is available in our GitHub repository. In there, you’ll find the documentation on how to run this project on your machine.

After having all the requirements set up on our computer, we can simply run make start to spin up our application with LocalStack. This will take a couple of minutes, but once it’s completed, we’ll have our API running inside the LocalStack container.

At this point, we can run the following command to see if our API is up and running.

We’ve confirmed that our API is up and running. Let’s see what Thundra Foresight can do now.

How to Integrate Foresight into LocalStack

If you have the latest version of LocalStack, all you need to do is set THUNDRA_APIKEY as an environment variable, and you are good to go with application monitoring. To enable Foresight as well, THUNDRA_AGENT_TEST_PROJECT_ID environment variable must be set.

You can get your API key and project ID at https://foresight.thundra.io.

For this project, you can set them in the Makefile. If that’s done, let’s run the tests with make test. After some time, we should be able to see the results.

Looks like our test has failed. Let’s check Foresight, and then click to our test to see if we can see what went wrong.

When you are on the test detail page, you can click the Trace Map button on the top right. It will open a window and automatically redirect you to your invocations trace map. Here, we can see the whole flow, identical to what we’ve designed at the beginning of this post. We will also be able to tell where the error comes from. If we click the JUnit 5 node on our map, we’ll see the whole trace chart for our test.

On the right-hand side of the screen, we see the original assertion error and its method marked as red on the trace chart. Let’s click to that second red row.

This opened our debugging window again. Now, let’s zoom in on that and play the execution until the end.

We can see that our getResponse list doesn’t have the FINISHED state as expected.

Now, we have some idea where our error originated and what variables were present at the time of the execution of our test. We call this Time Travel Debugging. With this instrumentation method, Thundra Agent will give you the ability to save your traces and play them line by line to debug your application and your tests.

Now, let’s see how we can fix this.

Chaos Is the Answer

Well, usually developers write tests to ensure their application doesn’t break when they develop new features, fix bugs or change anything in their codebase. Generally, this will give you a good estimate of how robust your application is. Not all errors come from our inability to write bug-free code, however.

Purposefully injecting a bug, an exception or latency might seem counterintuitive, but it’s better than the unexpected. This practice of doing a controlled experiment on your distributed system is called chaos engineering. At Thundra, we value the chaos, do our best to be prepared for the unexpected, and we want our customers to be the same, if not more. By using Thundra Agent in your project, you’ll have the capability to engineer your own chaos experiment for your applications and your tests.

So, that’s what we did here. If you go to the following file, you’ll see that we’ve implemented a class for chaos injection and injected an error to our Lambda #3.

To fix the error, we can simply comment out these lines. Let’s run the tests again and see our tests become successful.

In addition to application-level chaos engineering with Thundra, we can also inject errors at the infrastructure level with LocalStack, for example, injecting Kinesis stream errors by configuring KINESIS_ERROR_PROBABILITY=0.8. Let’s not go into too much detail here. We’ll cover that in a separate blog post.

Summing up: LocalStack users can now have a granular view of their tests by plugging in Thundra Foresight with the demonstrated steps.

You can sign up for Thundra and integrate your open source project for “free forever and with full functionality.”

Photo by Stacey Gabrielle Koenitz Rozells from Pexels.

Source: InApps.net

Rate this post
Read More:   Observing the Heterogenous Stack at Scale – InApps Technology 2022
As a Senior Tech Enthusiast, I bring a decade of experience to the realm of tech writing, blending deep industry knowledge with a passion for storytelling. With expertise in software development to emerging tech trends like AI and IoT—my articles not only inform but also inspire. My journey in tech writing has been marked by a commitment to accuracy, clarity, and engaging storytelling, making me a trusted voice in the tech community.

Let’s create the next big thing together!

Coming together is a beginning. Keeping together is progress. Working together is success.

Let’s talk

Get a custom Proposal

Please fill in your information and your need to get a suitable solution.

    You need to enter your email to download

      Success. Downloading...