Pennymac is (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market. 

At Pennymac, our people are the foundation of our success and at the heart of our dynamic work culture. Together, we work towards a unified goal of helping millions of Americans achieve aspirations of homeownership through the complete mortgage journey.

As a member of the Site Reliability Operations (SRO) team, you will help provide 24/7 monitoring and support of the company’s IT Infrastructure. Ideal candidates should have experience in Windows and Linux administration, in addition to experience working in AWS, as Pennymac is now almost completely migrated into the AWS cloud. Individuals in this role should be comfortable working in a fast-paced environment. Multitasking, in addition to communicating quickly and accurately, is critical to the success of anyone in this role.

The Site Reliability Engineer - Observability Operations will:

• Monitoring – Oversee 24/7 health monitoring of the company’s IT Infrastructure using tools such as AWS CloudWatch and New Relic. Drive observability maturity across the organization by identifying coverage gaps and implementing targeted improvements.
• Alert Management – Own the ongoing refinement of operational alerts. Implement advanced alerting rules and thresholds to proactively identify issues, reduce noise, and ensure every alert drives action.
• Observability Gap Analysis – Partner with Incident Management to identify monitoring and alerting gaps discovered during incident triage; prioritize and implement enhancements to prevent recurrence.
• App Team Engagement – Serve as an observability resource to application teams, assessing current instrumentation and providing actionable recommendations to improve monitoring maturity.
• Alert Quality Ownership – Lead initiatives to reduce alert noise, improve signal-to-noise ratio, and ensure every alert is actionable with clear runbook linkage.
• Operational Dashboard Development – Design and maintain operationally-focused dashboards in New Relic that support 24/7 triage, SLA tracking, and real-time incident response.
• Incident Management – Serve as an escalation point for complex incidents. Collaborate closely with the Incident Management team, Application Developers, Internal Support Teams, and 3rd Party Vendors to ensure timely and accurate resolution of service disruptions
• Advanced Systems Administration – Perform and troubleshoot a wide range of administrative tasks across Windows and Linux environments. Assist in optimizing system performance, conducting root-cause analyses, and implementing long-term fixes
• Virtual Server and Desktop Management – Handle more complex tasks associated with maintaining and troubleshooting the company’s virtual infrastructure. Provide guidance to junior engineers for routine issues
• Technical Troubleshooting and Investigation – Tackle advanced technical issues that are escalated from Engineer I/II. Conduct deep dives into infrastructure and application logs to pinpoint underlying problems
• Internal and External Escalation – Act as a liaison between multiple internal teams and external vendors for high-priority incidents. Ensure swift coordination and minimize downtime
• Change Management – Strictly follow and help refine the company’s established Change Management processes. Provide risk assessments and validation for proposed changes before approval
• Communication – Monitor and respond to incoming Calls, Chats, and Emails directed to the SRO team. Offer structured feedback to stakeholders when complex issues are underway
• Ticket Queue Management – Lead by example in managing multiple ticket queues (ServiceNow, JIRA, etc.). Take ownership of priority tickets and oversee distribution among the team
• Documentation – Maintain and expand the SRO team’s knowledge base. Author new Standard Operating Procedures (SOPs) that incorporate best practices gained from resolving advanced incidents
• Deployments – Coordinate and execute application and website code deployments using Jenkins, GitLab, or other CI/CD tools. Help optimize deployment workflows to reduce errors and downtime
• Data Backup and Compliance – Oversee backup tasks using CommVault, AWS Backup, and related tools. Ensure data retention meets or exceeds corporate and regulatory requirements
• Project Management – Drive or co-lead medium to large-scale projects related to infrastructure improvements, migrations, or optimizations. Collaborate with stakeholders to define scope, timelines, and resource needs
• Mentorship – Provide guidance to Engineer I and II staff on advanced troubleshooting methods, best practices in cloud administration, and effective incident response
  

• Bachelor’s Degree in Computer Science or comparable experience.
• Advanced AWS Certifications strongly preferred
• 3–5+ years of experience working in both Windows and Linux environments, with demonstrated success in advanced troubleshooting and administration.
• Hands-on experience with New Relic (dashboards, NRQL queries, alerting configuration)
• Demonstrated success improving monitoring coverage and alert quality
• Ability to consult with application teams on observability best practices.
• Strong analytical skills for identifying patterns in incident data
• Strong scripting or programming skills in PowerShell, Python, or a similar language; ability to automate repetitive tasks and streamline operations.
• Excellent organizational skills, with the ability to manage competing priorities and urgent issues in a fast-paced setting.
• Strong written and verbal communication skills; able to explain complex technical issues to stakeholders at various technical levels.
• Comfortable completing annual role-based training and certification assignments; dedicated to continual learning and development.
• Demonstrated ability to work independently on complex tasks and to collaborate effectively with cross-functional teams.

As one of the top mortgage lenders in the country, Pennymac has helped over 4 million lifetime homeowners achieve and sustain their aspirations of home. Our vision is to be the most trusted partner for home. Together, 4,000 Pennymac team members across the country are guided by our core values: to be Accountable, Reliable and Ethical in all that we do. 

Pennymac is committed to conducting a business that makes positive contributions and promotes long-term sustainable growth and to fostering an equitable and inclusive environment, where all employees and customers feel valued, respected and supported. 

Benefits That Bring It Home: Whether you're looking for flexible benefits for today, setting up short-term goals for tomorrow, or planning for long-term success and retirement, Pennymac's benefits have you covered. Some key benefits include: 

• Comprehensive Medical, Dental, and Vision
• Paid Time Off Programs including vacation, holidays, illness, and parental leave 
• Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)
• Retirement benefits, life insurance, 401k match, and tuition reimbursement 
• Philanthropy Programs including matching gifts, volunteer grants, charitable grants and corporate sponsorships
• We value the hard work and dedication of our employees. In addition to a competitive salary, positions may offer bonus opportunities.

To learn more about our benefits visit: 

https://pennymacnews.page.link/benefits

Compensation: Individual salary may vary based on multiple factors including specific role, geographic location / market data, and skills and experience as defined below:

• Lower in range - Building skills and experience in the role
• Mid-range - Experience and skills align with proficiency in the role 
• Higher in range - Experience and skills add value above typical requirements of the role 

Some roles may be eligible for performance-based compensation and/or stock-based incentives awarded to employees based on company and individual performance.