Pennymac logo
Principal Engineer AI
full-timeWestlake Village, Cary, Carrollton$90k - $150k

Summary

Location

Westlake Village, Cary, Carrollton

Salary

$90k - $150k

Type

full-time

Explore Jobs

About this role

PENNYMAC

Pennymac (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market. 

 

At Pennymac, our people are the foundation of our success and at the heart of our dynamic work culture. Together, we work towards a unified goal of helping millions of Americans achieve aspirations of homeownership through the complete mortgage journey.


Job Overview

As the Principal Applied AI Engineer, you will be the lead architect of the "cognitive" layer of
the AI platform. While the Backend Principal builds the deterministic infrastructure, you will
engineer the probabilistic systems that power our "Unified Context Library" and "Agentic
Orchestration" layer. You will design the autonomous workflows that turn a Product Manager’s
idea into a technical specification, and a User Story into deployable code.
This role sits at the intersection of elite software engineering and applied artificial intelligence.
You will not be training foundational models from scratch; rather, you will be mastering the art of
Applied AI: orchestration, retrieval-augmented generation (RAG), and the engineering of
agentic systems using the AWS Bedrock AgentCore framework. You will work within the AI
Platform Services division, a dedicated R&D unit tasked with delivering measurable velocity
improvements to the entire organization.
Why Join Pennymac?
● Architect the Future of SDLC: You will build the "Agent Factory" that drives our "conveyor
belt" of software delivery, moving us from a reactive to a proactive engineering culture.
● Greenfield Innovation: This is a rare chance to build an enterprise-grade AI platform from
the ground up, leveraging the latest in AWS Bedrock AgentCore and Agentic AI
frameworks.
● High-Impact & Visibility: Your work will directly impact the daily lives of hundreds of
engineers and product owners, reducing "Idea-to-MR" cycle times and eliminating manual
toil.

● Cutting-Edge Stack: Work with a modern, cloud-native stack (AWS, Node.js/TypeScript)
specifically tailored for high-performance AI applications.


A Typical Day

 

Architect Agentic Workflows
● Design and implement sophisticated multi-agent systems that can plan, execute, and
self-correct complex tasks (e.g., automated code reviews, test plan generation, and epic
decomposition).
● Develop robust orchestration flows using LangChain.ts and AWS Bedrock AgentCore,
defining how agents hand off tasks to one another and when to loop in humans for review.
● Engineer "hallucination checkpoints" and validation logic to ensure AI outputs are accurate,
secure, and deterministic where necessary.
● Implement the Model Context Protocol (MCP) to standardize how our agents interface
with internal tools like Jira, GitLab, and AWS infrastructure.

 

Build the Unified Context Library (RAG)
● Lead the strategy for our Retrieval-Augmented Generation (RAG) foundation. You will
design the pipelines that ingest, chunk, and vectorize institutional knowledge from
Confluence, Jira, and GitLab.
● Optimize Vector Database performance (e.g., Pinecone, Weaviate) and implement
advanced retrieval strategies (hybrid search, re-ranking) to ensure agents possess the
precise, domain-specific context needed for mortgage-tech tasks.
● Implement "memory" systems (Short-term and Long-term) that allow agents to retain
context across long-running sessions and provide personalized assistance to users.

 

AI System Engineering & Observability
● Design and maintain the Observability & Fine-Tuning Framework, ensuring we capture
every token, prompt, and user feedback signal (thumbs up/down) to systematically improve
agent performance over time.
● Define and enforce Prompt Engineering best practices, creating a reusable library of
system prompts that govern agent persona, tone, and output formatting.
● Build automated Evaluation Pipelines (using tools like LangSmith or custom harnesses)
to benchmark agent performance against "Golden Datasets" and prevent regression.

 

Technical Leadership
● Serve as the subject matter expert on Generative AI for the Platform Services division,
staying ahead of the curve on LLM capabilities, cost optimization, and model selection
(e.g., routing tasks between Claude 3.5 Sonnet, GPT-4o, and smaller, faster models).
● Mentor fellow engineers on the paradigm shift from deterministic coding to probabilistic AI
engineering.
● Drive the adoption of AI best practices across the wider organization.


What You’ll Bring

Must-Haves:
● Elite Engineering Core: Bachelor’s Degree in Computer Science or equivalent, with 8+
years of professional software engineering experience. You are a software engineer first,
who has mastered AI tools.
● TypeScript/Node.js Expert: Unlike most AI roles that focus on Python, our platform is built
on Node.js and TypeScript. You must have deep expertise in building backend services
and AI chains in this ecosystem.
● Applied AI & Agent Experience: Hands-on experience building applications powered by
LLMs. You have shipped products using frameworks like LangChain, Strands, or AWS
Bedrock.
● RAG Mastery: Proven track record of building production-grade RAG systems. You
understand the nuances of embeddings, vector stores (Pinecone, Milvus), and semantic
search.
● Cloud Native (AWS): Extensive experience with AWS serverless architecture (Lambda,
API Gateway, DynamoDB). Familiarity with AWS Bedrock and AgentCore is a significant
advantage.
● Systems Thinking: Ability to design complex, asynchronous systems where state is fluid
and outcomes are probabilistic.
● Startup Mentality: High ownership, high energy, and the ability to thrive in a fast-paced
"internal startup" environment.

 

 

Nice-to-Haves (Bonus Points):
● Experience with Evaluation Frameworks (e.g., LangSmith, Ragas) for automated testing
of LLM outputs.
● Familiarity with the Model Context Protocol (MCP) for standardizing AI tool connections.
● Background in Developer Tools (building CLI tools, IDE plugins, or CI/CD automations).
● Understanding of Graph Databases (e.g., Neo4j) for knowledge graph implementation
alongside vector search.


Why You Should Join

As one of the top mortgage lenders in the country, Pennymac has helped over 4 million lifetime homeowners achieve and sustain their aspirations of home. Our vision is to be the most trusted partner for home. Together, 4,000 Pennymac team members across the country are guided by our core values: to be Accountable, Reliable and Ethical in all that we do. Pennymac is committed to conducting a business that makes positive contributions and promotes long-term sustainable growth and to fostering an equitable and inclusive environment, where all employees and customers feel valued, respected and supported. 

 

Benefits That Bring It Home: Whether you're looking for flexible benefits for today, setting up short-term goals for tomorrow, or planning for long-term success and retirement, Pennymac's benefits have you covered. Some key benefits include: 

  • Comprehensive Medical, Dental, and Vision
  • Paid Time Off Programs including vacation, holidays, illness, and parental leave 
  • Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
  • Retirement benefits, life insurance, 401k match, and tuition reimbursement 
  • Philanthropy Programs including matching gifts, volunteer grants, charitable grants and corporate sponsorships

 

To learn more about our benefits visit: https://pennymacnews.page.link/benefits

 

For residents with state required benefit information, additional information can be found at: https://www.pennymac.com/additional-benefits-information

 

Compensation: Individual salary may vary based on multiple factors including specific role, geographic location / market data, and skills and experience as defined below:

  • Lower in range - Building skills and experience in the role
  • Mid-range - Experience and skills align with proficiency in the role 
  • Higher in range - Experience and skills add value above typical requirements of the role 

 

Some roles may be eligible for performance-based compensation and/or stock-based incentives awarded to employees based on company and individual performance. 

 


Salary

$90,000 - $150,000
Work Model

OFFICE

Other facts

Tech stack
Software Engineering,Applied AI,Node.js,TypeScript,AWS,RAG,Systems Thinking,Cloud Native,Agentic Systems,Observability,Prompt Engineering,Mentoring,Evaluation Frameworks,Graph Databases,Developer Tools,Machine Learning

About Pennymac

PennyMac Loan Services, LLC (NMLS #35953) is a top national mortgage lender with over 4 million lifetime customers. Whether you are new to the home loan process or an experienced buyer, Pennymac is dedicated to offering competitive rates and superior service.

Being an online mortgage lender means Pennymac can focus on the needs of its customers rather than maintaining a network of branches and banking products.

Our mission is to build a foundation of homeownership by enabling our customers to achieve and sustain their aspirations of home.

Pennymac Publication Rules
We A.R.E. Pennymac: Accountable, Reliable, Ethical. Pennymac does not pay or otherwise provide compensation for reviews/comments. All reviews/comments are subject to the disclaimer provided prior to posting and become the property of Pennymac once submitted. We reserve the right to validate, remove or republish your reviews/comments at any time.

Equal Housing Opportunity, PennyMac Loan Services, LLC, 3043 Townsgate Road, Suite 200, Westlake Village, CA 91361, 818-224-7442. NMLS ID # 35953 (www.nmlsconsumeraccess.org). For a complete listing of state licenses and important notices, please visit www.pennymac.com/state-licenses. Not all property types qualify. Some loan products may not be available in all states. Information, property type eligibility, rates and pricing are subject to change without prior notice at the sole discretion of PennyMac Loan Services, LLC. Ask your loan officer for details. © 2025 Private National Mortgage Acceptance Company, LLC, Pennymac and all related marks are trademarks of Private National Mortgage Acceptance Company, LLC and/or its subsidiaries or affiliates. Third-party content is the property of its respective owners. All rights reserved. (01-2025)

Team size: 1,001-5,000 employees
LinkedIn: Visit
Industry: Financial Services
Founding Year: 2008

What you'll do

  • You will architect and implement sophisticated multi-agent systems and design autonomous workflows that translate ideas into deployable code. Additionally, you will lead the strategy for Retrieval-Augmented Generation and ensure the performance of AI systems through observability and fine-tuning.

Ready to join Pennymac?

Take the next step in your career journey

Frequently Asked Questions

What does Pennymac pay for a Principal Engineer AI?

Pennymac offers a competitive compensation package for the Principal Engineer AI role. The salary range is USD 90k - 150k per year. Apply through Clera to learn more about the full compensation details.

What does a Principal Engineer AI do at Pennymac?

As a Principal Engineer AI at Pennymac, you will: you will architect and implement sophisticated multi-agent systems and design autonomous workflows that translate ideas into deployable code. Additionally, you will lead the strategy for Retrieval-Augmented Generation and ensure the performance of AI systems through observability and fine-tuning..

Why join Pennymac as a Principal Engineer AI?

Pennymac is a leading Financial Services company. The Principal Engineer AI role offers competitive compensation.

Is the Principal Engineer AI position at Pennymac remote?

The Principal Engineer AI position at Pennymac is based in Westlake Village, California, United States and Cary, North Carolina, United States. Contact the company through Clera for specific work arrangement details.

How do I apply for the Principal Engineer AI position at Pennymac?

You can apply for the Principal Engineer AI position at Pennymac directly through Clera. Click the "Apply Now" button above to start your application. Clera's AI-powered platform will help match your profile with this opportunity and guide you through the application process. You can also learn more about Pennymac on their website.