Specialized Analytics Senior Analyst

Job Description Overview The C12 is a developing professional role. Deals with most problems independently and has some latitude to solve complex problems. Integrates in-depth specialty area knowledge with a solid understanding of industry standards and practices. Good understanding of how the team and area integrate with others in accomplishing the objectives of the sub function/ job family. Applies analytical thinking and knowledge of data analysis tools and methodologies. Requires attention to detail when making judgments and recommendations based on the analysis of information. Typically deals with variable issues with potentially broader business impact. Applies professional judgment when interpreting data and results. Breaks down information in a systematic and communicable manner. Developed communication and diplomacy skills are required to exchange potentially complex/sensitive information. Moderate but direct impact through close contact with the businesses' core activities. Quality and timeliness of service provided will affect the effectiveness of own team and other closely related teams. TTS Analytics C12 What do we do? * The TTS Analytics team provides analytical insights to the Product, Pricing, Client Experience and Sales functions within the global Treasury & Trade Services business. The team works on business problems focused on enhancing client experience, driving acquisitions, cross-sell and revenue growth. * The team extracts relevant insights, identifies business opportunities, converts business problems into analytical frameworks, uses big data tools and AI/ML techniques to drive data driven business outcomes in collaboration with business and product partners. * The team works on building and operating Generative AI and deep learning solutions - Design, implement, and scale production‑grade AI applications end‑to‑end—from data ingestion and model services to user‑facing interfaces, observability, and secure deployments. ROLE DESCRIPTION The role will be AVP (C12) in the TTS Analytics team. The role will involve working on * As a key contributor to ideation on analytical projects to tackle strategic business priorities. * The role will require endless curiosity, as ambiguity and open-ended questions are a core part of the team’s work. * Own end‑to‑end delivery of AI‑powered products: requirements, design, implementation, testing, deployment, and support. * Design, build, and maintain automated, scalable evaluation pipelines and frameworks (e.g., using LLM-Eval-Harness, or custom internal frameworks). * Curate, version, and synthetically augment high-quality ground-truth ("golden") datasets covering diverse use cases, edge cases, domain-specific tasks, and adversarial vectors. * Implement metric suites evaluating task accuracy, semantic similarity, faithfulness, hallucination rates, toxicity, relevancy, bias, deterministic constraint satisfaction, and tool/agent execution fidelity. * Establish statistically sound LLM-as-a-Judge evaluation protocols, human-in-the-loop (HITL) calibration workflows, and deterministic/statistical baseline checks (BLEU, ROUGE, Exact Match, BERTScore). * Formulate automated test suites and migration runbooks to transition production workloads across model versions with minimal disruption and validated parity. * Develop structured, reproducible prompt architectures using advanced techniques (Few-Shot, Chain-of-Thought, ReAct, DSPy, Automated Prompt Optimization). * Analyze and optimize token budgets, prompt caching, model routing strategies (e.g., small SLMs for routing/extraction vs. large frontier models for complex reasoning), and speculative decoding strategies. * Enforce schema compliance (JSON schema, Pydantic, function calling / tool definitions) and deterministic constraints across varying foundation model backends. * Partner closely with platform architects, MLOps, backend engineers, and data engineers to integrate evaluation artifacts directly into CI/CD pipelines and deployment workflows. * Lead the AI governance process * Author comprehensive Object Development Documentation (ODD), Model Validation packages, model risk assessments, and technical artifacts detailing model design, limitation boundaries, training/evaluation data lineage, and testing methodologies. * Engage early with new AI initiative owners to scope evaluation criteria, acceptable error thresholds, bias assessments, and testing requirements prior to production onboarding. * Appropriately assess risk when business decisions are made, demonstrating consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency. * Ability to build partnerships with cross-function leaders. QUALIFICATIONS Experience: * Overall 8+ years of experience. 5+ years of Data Science/ML engineering experience, with at least 1–2 years dedicated to Generative AI, LLMs, NLP, and evaluation frameworks. Must have substantial experience in: · Strong proficiency in Python (async/concurrency, FastAPI/Pydantic, pytest) and standard data analysis stacks (Pandas, NumPy). · Hands-on experience with modern LLM tooling such as LangChain, LlamaIndex, Hugging Face, etc.. · Deep understanding of quantitative evaluation metrics (Semantic similarity, Faithfulness, Answer Relevancy, Precision/Recall, Hallucination scoring, Toxicity). · Experience building structured dataset curation pipelines, data cleansing, versioning (like S3), and synthetic data generation techniques. · Observability: application monitoring, centralized logging, tracing; strong debugging and problem‑resolution skills. · AI Governance Good to Have · Prior experience in financial services, healthcare, or heavily regulated enterprise environments, particularly with Model Risk Governance · Experience evaluating complex multi-step reasoning, tool-calling agents, memory systems, and Multi-Agent Orchestration frameworks. · Knowledge of data privacy and responsible AI practices; experience implementing audit and guardrail tooling. Education: * Bachelor / Masters (preferred) in Computer Science Engineering, Data Science, AI/ML, or related technical disciplines (or equivalent practical experience). This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required. \------------------------------------------------------ ## Job Family Group: Decision Management \------------------------------------------------------ ## Job Family: Specialized Analytics (Data Science/Computational Statistics) \------------------------------------------------------ ## Time Type: Full time \------------------------------------------------------ ## Most Relevant Skills Please see the requirements listed above. \------------------------------------------------------ ## Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. \------------------------------------------------------ Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law. If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi. View Citi’s EEO Policy Statement and the Know Your Rights poster.