Job DescriptionThe Data Integration team at OverDrive provides data for other teams to analyze and build their systems upon. We are plumbers, building a company-wide pipeline of clean, usable data for others to use. We typically don’t analyze the data, but instead we make the data available to others. Your job, if you choose to join us, is to help us build a real-time data platform that connects our applications and makes available a data stream of potentially anything happening in our business. Why Apply: We are looking for someone who can help us wire up the next step. Help us create something from the ground up (almost a green field). Someone that can help us move large data from one team to the next and come up with ideas and solutions around how we go about looking at data. Using technologies like Kafka, Scala, Clojure, F#. About You: You always keep up with the latest in distributed systems. You're extremely depressed each summer when the guy who runs highscalability.com hangs out the "Gone Fishin" sign. You’re humble. Frankly, you’re in a supporting role. You help build infrastructure to deliver and transform data for others. (E.g., someone else gets the glory because of your effort, but you don’t care.) You’re patient. Because nothing works the first time, when it comes to moving data around. You hate batch. Real-time is your thing. Scaling services is easy. You realize that the hardest part is scaling your data, and you want to help with that. You think microservices should be event-driven. You prefer autonomous systems over tightly-coupled, time-bound synchronous ones with long chains of dependencies. Problems You Could Help Solve: Help us come up with solutions around speeding up our process Help us come up with ideas around making our indexing better Help us create better ways to track all our data If you like to solve problems and use cutting edge technology – keep reading Responsibilities: Implement near real-time ETL-like processes from hundreds of applications and data sources using the Apache Kafka ecosystem of technologies. Designing, developing, testing and tuning a large-scale ‘stream data platform’ for connecting systems across our business in a decoupled manner. Deliver data in near real-time from transactional data stores into analytical data stores. R&D ways to acquire data and suggest new uses for that data. “Stream processing.” Enable applications to react to, process and transform streams of data between business domains. “Data Integration.” Capture application events and data store changes and pipe to other interested systems. Experience/Skills: Comfortable with functional programming concepts. While we're not writing strictly functional code, experience with languages like Scala, Haskell, or Clojure will make working with streaming data easier. Familiarity with the JVM. We’re using Scala with a little bit of Java and need to occasionally tweak the performance settings of the JVM itself. Familiarity with C# and the .Net framework is helpful. While we don’t use it day to day, most of our systems run on Windows and .Net. Comfortable working in both Linux and Windows environments. Our systems all run on Linux, but we interact with many systems running on Windows servers. Shell scripting & common Linux tool skills. Experience with build tools such as Maven, sbt, or rake. Knowledge of distributed systems. Knowledge of, or experience with, Kafka a plus. Knowledge of Event-Driven/Reactive systems. Experience with DevOps practices like Continuous Integration, Continuous Deployment, Build Automation, Server automation and Test Driven Development. Things You Dig: Stream processing tools (Kafka Streams, Storm, Spark, Flink, Google Cloud DataFlow etc.) SQL-based technologies (SQL Server, MySQL, PostgreSQL, etc.) NoSQL technologies (Cassandra, MongoDB, Redis, HBase, etc.) Server automation tools (Ansible, Chef, Puppet, Vagrant, etc.) Distributed Source Control (Mercurial, Git) The Cloud (Azure, Amazon AWS) The ELK Stack (Elasticsearch, Logstash, Kibana) What’s Next: As you’ve probably guessed, OverDrive is a place that values individuality and variety. We don’t want you to be like everyone else, we don’t even want you to be like us—we want you to be like you! So, if you're interested in joining the OverDrive team, apply below, and tell us what inspires you about OverDrive and why you think you are perfect for our team. OverDrive values diversity and is proud to be an equal opportunity employer.