The most common reason (if not the only) for alerts is deploying new code to production APIs.
It’s always the new code that makes all the noise and wakes you up in the middle of the night — almost like a new born baby. By contrast, old code sits quietly in production minding its own business. If your production environment has been humming like a peaceful symphony for the past week, the chance that it will start throwing alerts in the next hour are slim to none.
So how do you make sure that your new code doesn’t break your production symphony?
If you are familiar with Diffy, you already know that comparing the behavior of your new code to your old code against production traffic is the best guarantee you can ask for.
Over the years, Diffy has helped many large tech-heavy engineering teams achieve their quality aspirations. While Diffy users continue to get tremendous value from Diffy’s UI, our larger enterprise customers have been seeking better integration with their existing observability stack.
We did this custom integration for some customers but always wanted something like OpenTelemetry to come along so we would no longer have to build and maintain custom integrations.
There are 3 aspects to observability and this is how Diffy leverages all of them via OpenTelemetry:
Logging
Diffy logs include important lifecycle events as well as traffic events for every sampled request sent to Diffy. These can now be seamlessly ingested into your favorite sink.
Metrics
One of the most valuable things Diffy does is shred your traffic schema to analyze and aggregate differences observed at various nodes. This creates an interesting opportunity for you to receive schema-based metrics from Diffy.
Tracing
Lastly, Diffy now exports traces to your favorite tracing backend for you to inspect all sorts of interesting metadata. e.g. The Diffy UI limits the number examples it shows to carefully manage storage constraints — you can now observe the request and all 3 responses within the traces.
The logs, metrics, and traces currently exported by Diffy are based on a survey of our existing users. Please feel free to open an issue and let us know if there is anything else you need in any of these dimensions.
Try it out! Checkout our Advanced Deployment guide to explore these cool dashboards for yourself. If you have any questions, please feel free to reach out to us for support at isotope@sn126.com.