In production, we have to change the table structure to address new business requirements. The table schema has to change to add/delete/rename table columns. Any of these changes affect downstream ETL jobs adversely. In order avoid these, we have to make corresponding changes to ETL jobs and target tables.
Schema evolution allows you to update the schema used to write new data while maintaining backwards compatibility with the schemas of your old data. Then you can read it all together as if all of the data has one schema. Please read more on Avro serialization at the following URL: https://avro.apache.org/. In the following example, I will demonstrate how Avro ...
