Apache Avro is a data format that combines the readability of JSON and the compression of binary data serialization. Simply – models structure is sent in the header, but data itself is well-compressed to bytes.
You can find a serializer implementation with a very intuitive interface inspired by Netwonsoft.Json here:
https://github.com/AdrianStrugala/AvroConvert
Now, I would like to share with you an idea of how to speed up and decrease network usage by changing the data format in the REST API communication. It can reduce the response time to up to 30%. In fact in .NET Core, you just need 2 classes: custom InputFormatter and OutputFormatter.
Implementation details and serialization benchmark could be found in the article: https://xabe.net/why-avro-api-is-the-best-choice/
I’m waiting for your questions or comments!
Avro isn’t nearly as readable as JSON. The schema is ‘readable’, sure, but when the data is machine-readable binary garbage, that doesn’t help you.
You claim this is ‘more secure’, but security through obscurity is a bad idea.
Each of them speed’s up the response for about 25-30%.
Not by your benchmarks, it doesn’t. You can’t count only serialization time, you have to take deserialization into account as well. You did this in your benchmark, but it clearly shows that JSON is the winner, with a round-trip time of 96 + 138 = 234 ms versus 82 + 200 = 282 ms for the best case for Avro.
I ran a combined benchmark (serialize, then deserialize), which still shows Avro is the winner:
This was done using BenchmarkDotNet, which not only supports many different benchmarking scenarios, but also makes sure benchmarking is done right, and it collects a whole boatload of interesting statistics, such as memory allocation. As you can see, AvroConvert uses about 8 times as much memory as System.Text.Json does.
Your comparison includes Gzip as a compression format for JSON, but it doesn’t mention what the performance of that is. And as for efficiency, why didn’t you include something like Brotli? That compression format is especially well-suited to on-the-fly compression, and it achieves nice ratios. In my testing, Brotli achieved about 48% compression versus 42% for Gzip.
For that matter, why didn’t you look into something established, like MessagePack or Protobuf?
Brotli has another advantage: it’s integrated into HTTP. You simply tack on a Accept-Encoding: br header to your request and, assuming the web server supports it, you get a Brotli-encoded response with a Content-Encoding: br header. The nice thing about this is that most tools understand Brotli, so you can have your cake and eat it too: data is compressed on the wire, but when you view it in something like Postman, it is automatically decompressed to something human-readable.
As for performance, Brotli increases the time taken, but it reduces the memory consumption, which is not the case for AvroConvert, when you enable compression:
Lastly, you kind of strangle your premise by listing how long it takes to transmit something over a 100 mbit/s connection. That’s not a very realistic scenario if you’re talking about server applications. Most servers (even virtual ones) have a data link of 1000 or even 10000 mbit/s.
For that matter, why didn’t you look into something established, like MessagePack or Protobuf?
This. If you’re gonna go binary, you need to compare with existing binary options.
Fixed formatting.
Hello, LetMeUseMyEmailFfs: code blocks using triple backticks (“`) don’t work on all versions of Reddit!
Some users see this / this instead.
To fix this, indent every line with 4 spaces instead.
FAQ
You can opt out by replying with backtickopt6 to this comment.
C# devs
null reference exceptions