Data Engineer (Gen AI)

Interactive Brokers Group, Inc. (Nasdaq: IBKR) is a global financial services company headquartered in Greenwich, CT, USA, with offices in over 15 countries. We have been at the forefront of financial innovation for over four decades, known for our cutting-edge technology and client commitment.

IBKR affiliates provide global electronic brokerage services around the clock on stocks, options, futures, currencies, bonds, and funds to clients in over 200 countries and territories. We serve individual investors and institutions, including financial advisors, hedge funds and introducing brokers. Our advanced technology, competitive pricing, and global market help our clients to make the most of their investments.

Barron's has recognized Interactive Brokers as the #1 online broker for six consecutive years. Join our dynamic, multi-national team and be a part of a company that simplifies and enhances financial opportunities using state-of-the-art technology.

The Enterprise Architecture organization is looking for a Data Engineer to join the team and help build the next generation of data infrastructure and AI-enabled workflows. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines, data lake platforms, and analytics solutions that support enterprise-wide AI initiatives and advanced analytics capabilities. You will partner closely with internal development teams and IT leadership to architect data solutions that meet diverse use-cases across the organization. This role offers the opportunity to work with cutting-edge data technologies, AI knowledge bases, and cloud-native data platforms as we scale our data operations. Your work will focus on delivering robust, well-documented data solutions and establishing best practices for data engineering across the enterprise.

· Design, build, and maintain scalable data crawlers and ETL/ELT pipelines to ingest data from various sources including enterprise collaboration tools, web applications, and internal databases

· Develop and manage data lake platform infrastructure including S3-based storage, Iceberg tables, AWS Glue data catalog, and cloud data warehouse solutions for analytics workloads

· Build and optimize real-time streaming data pipelines using Kafka for event-driven analytics and data processing

· Create and maintain data transformation pipelines to clean, curate, and prepare data for analytics and AI/ML applications, supporting multiple data formats including structured, semi-structured, and unstructured data (text, images, audio, video)

· Develop Python-based applications for data ingestion, processing, and integration supporting Gen AI RAG workflows and knowledge base systems

· Collaborate with internal development teams, data scientists, and stakeholders to understand requirements, architect appropriate data solutions, and create comprehensive technical documentation

· Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and data quality

· Write clean, maintainable, well-tested code following software engineering and data engineering best practices

· Create comprehensive technical documentation for data pipelines, architectures, and platform capabilities

· 6+ years of hands-on data engineering experience with modern data stack technologies

· Strong experience with AWS cloud services, particularly: S3, AWS Glue, Athena, EMR, Lambda

· Proficiency in Python for data processing, ETL, and application development

· Experience with PySpark on EMR for large-scale data processing

· Strong SQL skills for data analysis and transformation

· Experience building and maintaining ETL/ELT pipelines at scale

· Experience with Kafka for streaming data pipelines and real-time data processing

· Knowledge of data lake architectures and modern table formats (e.g., Iceberg)

· Experience with CI/CD practices using Git, version control systems, and containerization (Docker)

· Understanding of data modeling, data warehousing concepts, and analytics best practices

· Exceptional problem-solving and analytical skills

· Excellent collaboration and communication (verbal and written) skills

· Self-motivated with ability to work independently and manage multiple priorities

· Willingness and enthusiasm to learn AI/ML technologies, stay current with emerging data engineering trends

• Self-motivated and able to handle tasks with minimal supervision.
• Superb analytical and problem-solving skills.
• Excellent collaboration and communication (Verbal and written) skills.
• Outstanding organizational and time management skills.