Personalization’s Hulk squad produces Human Understandable Language Knowledge to enrich content understanding. We utilize Large Language Models to understand podcasts and audiobooks, building reliable, scalable systems to distribute that knowledge to Spotify internal teams, users, and creators. We are looking for a Data Engineer with interest in ML techniques to join our team and help build the future of podcast and audiobook listening experiences for millions of listeners at Spotify. This is a unique opportunity to help develop and shape Spotify recommendations. You’ll grow your skills in engineering at scale, work with a cross-functional team of Machine Learning Engineers, Backend Engineers, Data Engineers, and researchers, and join a super motivated and supportive team.
\n
What You'll Do- Work with large-scale data pipelines using data processing frameworks like Scio (built on Apache Beam), Hendrix ML, Ray, PyTorch, and Dataflow and other Google Cloud Platform offerings
- Support streaming and batch inference of core LLM models
- Develop new data pipelines to meet emerging needs and maintain our existing data systems as necessary to ensure they are reliable, scalable, and well tested
- Work with ML engineers to develop training and evaluation datasets for Generative LLM models
- Own the development of data solutions from architecture to delivery
- Collaborate with other backend and data engineers, ML engineers, data scientists, researchers, and other stakeholders
- Develop monitoring tools to automatically track AI product quality. This likely involves the use of LLMs to evaluate results as well as closer monitoring of performance indicators, and creation and maintenance of metrics datasets and dashboards
- Work in an agile team to continuously experiment, iterate and deliver on new product objectives
- Share knowledge, promote standard methodologies, making your team the best version of itself through mentorship and constructive accountability
Who You Are- You are passionate about data and proficient in data modeling, data access, and data storage optimization
- You have experience building high-scale production data pipelines with one or more data processing frameworks such as Scio (built on Apache Beam), and major props if you know Ray
- You might have worked with orchestration frameworks such as Flyte
- You care about agile software processes, data-driven development, reliability, and responsible experimentation
- You value collaboration and partnership within teams
- Experience, exposure, or curiosity about ML Data processing and inference pipelines, LLM fine-tuning, and/or using LLMs for data analysis
- Bonus if you have experience with NRT (near real time) systems
- Bonus if you experience with prompt engineering for LLMs and prompt optimization tools like DSPy
Where You'll Be- We offer you the flexibility to work where you work best! For this role, you can be within the North America region as long as we have a work location.
- This team operates within the Eastern Standard time zone for collaboration.
\n
The United States base range for this position is $160,091- $228,702 plus equity. The benefits available for this position include health insurance, six month paid parental leave, 401(k) retirement plan, monthly meal allowance, 23 paid days off, 13 paid flexible holidays, paid sick leave. These ranges may be modified in the future.
Spotify is an equal opportunity employer. You are welcome at Spotify for who you are, no matter where you come from, what you look like, or what’s playing in your headphones. Our platform is for everyone, and so is our workplace. The more voices we have represented and amplified in our business, the more we will all thrive, contribute, and be forward-thinking! So bring us your personal experience, your perspectives, and your background. It’s in our differences that we will find the power to keep revolutionizing the way the world listens.
At Spotify, we are passionate about inclusivity and making sure our entire recruitment process is accessible to everyone. We have ways to request reasonable accommodations during the interview process and help assist in what you need. If you need accommodations at any stage of the application or interview process, please let us know - we’re here to support you in any way we can.
Spotify transformed music listening forever when we launched in 2008. Our mission is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art and billions of fans the chance to enjoy and be passionate about these creators. Everything we do is driven by our love for music and podcasting. Today, we are the world’s most popular audio streaming subscription service.
