Jason Heiss

This is the complete version of my resume. An abridged version which lists only the highlights of my work experience is at http://aput.net/~jheiss/resume.php?abridge.

Introduction

I am an experienced manager, Unix, security and networking systems engineer and consultant with experience in organizations ranging from large corporations and Universities to small Internet startups. I have led teams deploying and managing massive, complex environments. I have experience building secure, scalable and manageable computing systems from the ground up as well as re-engineering existing systems to better meet the needs of the organization.

Experience

April 2008 to Present

Infrastructure Architect, Yellowpages.com (a division of AT&T)

Design and implement tools and processes to automate the deployment and management of the operations infrastructure which includes Linux, Solaris and Windows servers, SAN and NAS storage, and networking infrastructure.

• Design and implement an operations database responsible for:
  • Tracking assets (servers, storage devices, network devices, etc.)
  • Storing configuration of users, networks, etc. and feeding that configuration data to the Puppet system configuration management tool.
  • Managing the state of OS installs and re-installs and feeding that data to Kickstart.
• Design and implement improvements to the Puppet system configuration management environment to make it more robust, scalable and accessible.

December 2006 to April 2008

Manager of System Administration, Yahoo!

Manager of the 15+ person System Administration team. My team is responsible for the deployment and maintenance of the Advertiser and Publisher group's 15,000+ production systems which run the Sponsored Search (text) and Display (graphical) advertising applications and generate a significant portion of Yahoo!'s revenue. My team includes employees in Burbank, California, New York City, Tokyo, Japan and Dublin, Ireland who help provide round-the-clock coverage for our environment.

In addition to previous responsibilites:

• Lead the team during the Apex project to integrate the text and display advertising businesses within Yahoo!, merging systems managed by three different teams.
• Integrate the four member SA team from the Right Media acquisition.
• Lead the team to develop proactive tools and processes for managing our environment. A complete self-service user management system allowing our customers to manage the accounts and sudo privileges for hundreds of different applications while maintaining compliance with the approval processes required by internal security teams and Sarbanes-Oxley. Extensive hardware monitoring with automatic generation of tickets to the on-site Data Center Operations teams for hardware repairs. And numerous other tools and processes to ensure we catch issues before they turn into problems.

May 2005 to Decmber 2006

Manager of System Administration Operations, Yahoo! Search Marketing

Manager of the 8 person System Administration Operations team. My team is responsible for the deployment and maintenance of the Search Marketing group's 10,000+ production systems. My team includes two international employees (Tokyo, Japan and Dublin, Ireland) who help provide round-the-clock coverage for our environment.

• Led the team through the growth of the environment from 3,000 to 10,000+ servers with minimal staffing increases. At one point we deployed more than 4,000 servers in one month.
• Led the team through the integration of the Overture/YSM and Yahoo! environments, gradually bringing the YSM environment into compliance with Yahoo! standards and practices.
• Established a strong working relationship between the YSM SA team and three other SA teams within Yahoo!
• Led the team through the Panama project, a year-long complete overhaul of both the systems and applications for the entire YSM environment.
• Managed a 24x7x365 on-call schedule with multiple tiers and 15+ particpants.
• Coordinated the interview and hiring process for multiple employees. Wrote year-end reviews for team and made recommendations for salary, bonus and equity compensation adjustments.
• Ensured team compliance with SLAs in handling our ticket queue. The SA team consistently maintained one of the best rates of SLA compliance within YSM. In conjuction with the YSM Reporting team I pioneered the development and use of quarterly reports on individual effectiveness in responding to ticket requests, which we used to reward our top performers and mentor lower performers. I also oversaw and monitored team compliance with YSM change management practices.
• Coordinated team compliance with Sarbanes-Oxley requirements. Met with internal and external auditors on a regular basis to discuss our practices and procedures with regards to account provisioning and other security practices.

April 2004 to May 2005

Senior System Administrator, Yahoo! Search Marketing (formerly Overture)

Member of the 15+ person System Administration team within Production Operations, responsible for the operation of the 3000+ computer systems in YSM's production operations. Manage a mixture of Red Hat Linux, FreeBSD and Solaris systems providing hundreds of millions of advertisements daily for YSM's partners, including Yahoo!, CNN and MSN.

• Progressed to the architecture team, and then to team lead within 6 months of being hired.
• Responsible for the planning and execution of the rollout of Red Hat Enterprise Linux to YSM's Linux systems.
• Led the deployment of over 500 new servers.
• Responsible for the planning and execution of an upgrade from a poor performing configuration management system to a more modular, easier to manage, higher performance system for the management of YSM's production systems. I designed and wrote this system (see Accomplishments section).

July 2001 to April 2004

Senior Computer Specialist, Electrical Engineering Department at the University of Washington

System administrator for the Electrical Engineering department at the University of Washington. Responsible for maintaining a mixture of Linux and Solaris servers and workstations. Services provided include mail (SMTP, S/IMAP, S/POP), NFS, Kerberos, LDAP, NIS, Samba, print services, DNS, DHCP and HTTP/HTTPS.

• Coordinated roll-out of Red Hat Enterprise Linux. Upgraded all of our Linux servers with little user interruption or downtime.
• Deployed the configuration file management system I designed (see Accomplishments section). This system allowed us to upgrade our entire Linux infrastructure to Red Hat Enterprise Linux in less than a month with minimal user downtime and is still used for ongoing maintenance of all department Linux and Solaris machines.
• Designed and configured Kerberos 5 and LDAP services for providing user authentication and Unix user information. Transitioned department computers from a mixture of NIS and copied passwd and group files to the new services. Integrated Kerberos and LDAP into centralized user management tools.
• Trained other department system administrators in the use of Kerberos and LDAP.
• Configured load balancer to provide greater fault tolerance for LDAP service.
• Secured printers in public labs against unauthorized printing.
• Implemented dynamic DNS with TSIG authentication for the DHCP servers run by the department.
• Restored various backup systems (Amanda, BRU and others) to operational status and documented associated procedures (configuration, restores, and tape management and retention).
• Implemented a two-tier backup system for user data: The first tier system uses rsync to copy data to a remote system and creates daily snapshots. The second tier uses traditional tape archives. This allows for data growth while reducing the need for expensive new tape hardware. It also greatly improves the time it takes to perform a restore, as most restores can be done quickly from the remote server instead of tape.
• Configured JumpStart and Kickstart services for automated installations of Solaris and Red Hat Linux.
• Co-managed approximately 30 servers.
• Managed a computing lab consisting of 20 SunRay network appliances and an associated E450 server, as well as 20 SunBlade 100 workstations and two associated SunBlade 1000 servers.
• Documented all common procedures associated with system administration within the EE computing environment.
• Restored Big Brother server monitoring system to operational status, upgraded to latest version, made sure all servers were monitored, and wrote scripts to simplify administration.
• Designed and implemented a web-based user management system. System replaced several disjointed text and web-based systems.

June 1997 to July 2001

As a Consultant with Collective Technologies, I worked on numerous projects at a variety of clients.

• AT&T Wireless Services  (A leading provider of wireless communications services)  May-July 2001
• Configured SunScreen SecureNet firewall software on 38 systems (configured in 14 clusters using Veritas Cluster Server) to protect them from network attacks.
• Trained client personnel on maintenance and operations procedures for SunScreen software.
• Neptune.com  (Internet startup)  February 2001
• Rebuilt and secured four Linux servers that had been broken into by crackers.
• Designed, built, and installed a Linux-based, packet filtering firewall using a screened-subnet (DMZ) architecture.
• Instructed client personnel on best security practices.
• Evaluated Pivio network appliance for possible use as a firewall in the client's infrastructure.
• Aventail  (Working on Aventail.Net, an Internet startup providing a managed VPN service)  February 2000-February 2001
• Designed and built multiple Data Centers and Network Operations Centers in support of a managed security service.
• Configured internal and external DNS, mail (Sendmail and SSLified IMAP), and NTP services.
• Designed and implemented a network monitoring system using Big Brother.
• Provided NFS and NIS services for NOC and Service Development.
• Provided multiple FreeBSD and Linux-based packet filtering firewalls.
• Configured Samba, integrating it into an existing Windows domain, to provide file services to Windows desktop systems.
• Configured Veritas NetBackup to backup 100 GB FreeBSD file server and other systems.
• Configured Cisco routers.
• Updated the JumpStart configuration to comply with best practices and to facilitate future maintenance.
• Built numerous Solaris packages, and wrote extensive shell (Bourne and Korn) and Perl scripts to automate JumpStart and operations tasks.
• AT&T Wireless Services  (A leading provider of wireless communications services)  May 1999-December 1999
• Audited production firewall configuration and support organization, and recommended changes to improve security.
• Wrote requirements documents for security-related improvements to four products from Ericsson and Lucent.
• Performed security audits on a Tandem Unix server, a Sun server, and an IBM RS/6000 running AIX. Audited operating systems and TCP/IP network configurations, and recommended changes to improve security.
• Assisted with upgrading the CDPD (Cellular Digital Packet Data) Division's network management system from SunNet Manager to Solstice Enterprise Manager.
• Intuit  (The makers of Quicken, TurboTax and other financial software)  April 1999
• Updated JumpStart configuration to comply with best practices.
• Simplified JumpStart scripts to facilitate future maintenance.
• Added features to the local JumpStart configuration script to support a wider variety of hardware and software configurations.
• Documented changes and trained Intuit personnel in using the modified configuration.
• Vixel Corporation  (A leading supplier of Fibre Channel interconnect products)  April 1999
• Provided operational support for Sun SPARC Enterprise servers.
• Lucent Technologies  (A leading Internet infrastructure provider)  March 1999
• Evaluated a UUCP configuration problem on Sun SPARC Enterprise servers.
• AT&T Wireless Services  (A leading provider of wireless communications services)  June 1998-January 1999
• Supported production and development environments for the Technical Information Department.
• Supported two Sun E4000s, UltraSPARC 2 and UltraSPARC 1 servers, and SPARCstation systems.
• Provided networking support using a Cisco Catalyst 5000 switch.
• Installed and configured Sun hardware and the Solaris operating system.
• Maintained the existing Legato NetWorker backup system.
• Migrated production backups to ADSM.
• Coordinated the move of production servers to a new site.
• Installed, configured, and supported RAID 0+1 and RAID 5 SPARCstorage arrays on production and development servers using Veritas Volume Manager.
• Configured and maintained software packages to monitor servers and the network.
• Supported Sun desktop SPARCstations.
• Configured a secure Web server on Sun hardware running Solaris, and set up the server outside the company firewall.
• Configured a secure Solaris system, and installed Tripwire to monitor the security status.
• Configured and maintained ssh (Secure Shell) for remote logins to servers.
• ADC/Kentrox  (A leading manufacturer of WAN networking hardware)  April 1998
• Updated a Solaris JumpStart image to the latest patch cluster, and fixed, upgraded, or originally wrote custom configuration scripts as part of the JumpStart image.
• Automated printer configuration on SunOS and Solaris clients.
• Configured new printer queues, using HP JetAdmin software.
• Assisted in the move of 70 engineers and their workstations to a new building.
• Performed day-to-day administration and problem resolution for SunOS and Solaris workstations.
• Modified a backup script to support machines with multiple tape drives.
• Hughes Space and Communications  (The world's leading satellite systems provider)  June 1997-March 1998
• Automated and performed remote installation of Solaris and patches to over 500 workstations in eight hours.
• Previous rollouts had taken six months.
• Wrote Perl scripts to ensure that NIS maps were correct and hardware requirements were met, and to perform installations, using JumpStart.
• Trained system operators to use these scripts.
• Configured a Netra NFS server, and upgraded memory and disk hardware.
• Configured backups, using Solstice Backup with a DLT tape stacker.
• Configured Syntax TotalNET Advanced Server 4.x and 5.x as a file service for Windows NT and Macintosh clients.
• Supported two Sun servers running uShare 4.x for file service to Macintosh clients.
• Configured backups to 8mm tape stackers.
• Added APC UPSs, and configured PowerChute software.
• Set up and maintained a Netscape Enterprise Server 2.x Web server, including virtual hosting, using Sun's Netra Internet Server software package for the primary Intranet Web server.
• Developed a database system to keep track of operating system revisions and other hardware and software information for client workstations.

May 1995 to June 1997

As a Webmaster and Undergraduate House Computer Representative for the Campus Computing Organization at the California Institute of Technology:

• Performed Web and CGI development, and maintained the main Web pages for the California Institute of Technology.
• Installed, configured, and maintained Netscape 1.x and Apache 1.x Web Servers.
• Set up and administered Linux on several PCs and a Sun workstation, using Red Hat and Slackware distributions.
• Provided maintenance on disk drives, installed memory upgrades, and supported other PC hardware.

Education

• Bachelor of Science in Biology, California Institute of Technology (Caltech)
• Introduction to Cisco Router Configuration, Advanced Cisco Router Configuration courses
• USENIX LISA conference and tutorials in 1999, 2002, 2003, 2004, 2005 and 2006.
• USENIX BSDCon conference in 2000, including Marshall Kirk McKusick's BSD Internals tutorial
• Murray Timber Framing 2 day workshop

Accomplishments

• Presented a paper titled "Enterprise Rollouts with JumpStart" at the LISA '99 conference in Seattle, WA
• Active member of the Open Source community.
  • Author of Distributor, a software load balancer, and other applications
  • Frequently contribute patches and other suggestions to the maintainers of Open Source projects
• Designed and implemented a system for managing configuration files on Unix systems. Each configuration file is controlled by an XML file that specifies how the file is built on each machine as well as metadata settings like permissions and ownership. This system has been used at several of my employers to manage over 10,000 production systems.
• Eagle Scout, Boy Scouts of America