This event has ended. Visit the official site or create your own event on Sched.
Thank you for participating in Big Data Tech 2018! Access conference presentation decks here.
Tuesday, June 5 • 10:45am - 11:15am
The Future of ETL (isn't what it used to be)

Log in to save this to your schedule and see who's attending!

Feedback form is now closed.
Gwen Shapira will share design and architecture patterns that are used to modernize data engineering. We will see how Apache Kafka, microservices and event streams are used by modern engineering organizations to efficiently build data pipelines that are scalable, reliable and built to evolve.

We'll start the presentation with a discussion of how software engineering changed in the last 20 years - focusing on microservices, stream processing, cloud and the proliferation of data stores. These changes represent both a challenge and opportunity for data engineers.

Then we'll present 3 core patterns of modern data engineering:

  • Building data pipelines from decoupled microservices
  • Agile evolution of these pipelines using schemas as a contract for microservices
  • Enriching data by joining streams of events

We'll give examples of how these patterns were used by different organizations to move faster, not break things and scale their data pipelines. We'll also show how these can be implemented with Apache Kafka.

avatar for Gwen Shapira

Gwen Shapira

Principal Data Architect, Confluent
Gwen is a principal data architect at Confluent helping customers achieve success with their Apache Kafka implementation. She has 15 years of experience working with code and customers to build scalable data architectures, integrating relational and big data technologies. She currently... Read More →

Tuesday June 5, 2018 10:45am - 11:15am
K1450 (Fireside Room) Normandale Partnership Center, 9700 France Ave So, Bloomington, MN 55431