Sn126 is proud to announce a major update to Diffy. But a quick history recap on how we got here.
Diffy was originally built at Twitter in 2014. That's almost 8 years ago! Just let that sink in. There were many technology decisions that were simply the result of being born at Twitter. e.g.
- Diffy was written in Scala.
- It used Finatra for API design and Finagle for networking.
- It was built to test Thrift APIs before it was generalized to test HTTP APIs.
- Twitter had just introduced Manhattan - Twitter's proprietary key-value store of choice. Any new services being built were using Manhattan.
Many of these technology choices were innately great. The core difference engine written in Scala could not and still can't be written as concisely in Java. Finagle offered a beautiful abstraction for networking that made it possible to implement and merge http support in a matter of days.
After open sourcing Diffy, we began receiving support requests from other engineering teams. We began to understand the world outside had much more diverse requirements. This eventually lead to the birth of Sn126.
To meet the new set of requirements from our customers, we decided to build a more advanced version of Diffy called Isotope. This was the time when the tech choices we had made at Twitter started turning into tech debt. Scala and Finagle could not catchup to Java and Spring Boot. Scala's build story never got any better and simple tasks like updating our Finagle dependencies ended up becoming time consuming work.
Meanwhile, most of our customers were using these other technologies along with Nodejs and Python. As we worked closely with their engineering teams, we experienced their best practices and started adopting them ourselves. Every time we would ship minor changes to Diffy, we would ponder what it would be like to revamp it with the most cutting edge tech.
Finally, the last straw was the uncertainty around Twitter acquisition and the damage it did to the stability of its open source technologies. With my former colleagues not being sure about having a job in a few months, how could I be sure about the maintenance and support of their open source code at Twitter.
As the case went to trial, we started working on all the changes we wanted to free ourselves from our accumulated tech debt and rewrite significant chunks of Diffy.
- We swapped out Finagle with Spring Boot and introduced significant new chunks of Java.
- We added MongoDB integration to eliminate all the memory pressure caused by keeping all the difference results in-memory.
Why did Diffy keep everything in-memory? Remember the story about Manahattan from earlier ... if Diffy were to have any storage, it would have to be Manhattan ... and you can't open-source something with a proprietary dependency ... so we decided not to have any storage at all!
- We rebuilt the UI from scratch using Typescript, React and implemented material design.
Coincidentally, by the time the Twitter-Musk deal closed on October 28th, we had already removed every Twitter open-source dependency from the code base.
There are many great features we have built in Isotope that we can now introduce to Diffy. We have already added request transformation capabilities that allow production requests to be sanitized and rewritten before being multicast to test targets. We will talk about that and others in the next post.
In the meantime, feel free to take the new Diffy out for a spin and do share your feedback. We are isotope@sn126.com.