Introduction
I am an experienced manager, Unix, security and networking systems
engineer and consultant with experience in organizations ranging from
large corporations and Universities to small Internet startups. I have
led teams deploying and managing massive, complex environments. I have
experience building secure, scalable and manageable computing systems
from the ground up as well as re-engineering existing systems to better
meet the needs of the organization.
Experience
AT&T Interactive: April 2008 to Present
During my time at AT&T Interactive I progressed through several
positions of increasing responsibility.
May 2010 to Present, Executive Director,
System Operations,
AT&T Interactive / Yellowpages.com
Lead the Unix System Administration, Windows System Administration, and
Database Administration teams. Responsible for the operation of a
production environment supporting a billion dollar Internet advertising
business.
April 2008 to April 2010, Director, Platform Services,
AT&T Interactive / Yellowpages.com
Lead a team of architects and tools developers who set architectural
direction and implement tools and processes to make our production
environment more robust, scalable, agile and efficient. Responsible for
the architecture of a complex, distributed computing environment
supporting a high traffic Internet site. Automate the deployment and
management of the operations infrastructure which includes Linux,
Solaris and Windows servers, SAN and NAS storage, Oracle, MySQL and
PostgreSQL databases, and networking infrastructure. Design and
deploy private cloud and grid computing environments for use by multiple
teams within the organization.
- Design and implement an operations database
- Released as Open Source:
http://nventory.sourceforge.net/
- Responsible for:
- Tracking assets: servers, storage devices, network devices,
etc.
- Storing configuration of users, networks, etc. and feeding
that configuration data to the system configuration management
tool.
- Storing the association of services to hardware assets and
the relationships between services
- Design and implement an updated version of my system configuration
management tool
- Released as Open Source:
http://etch.sourceforge.net/
- Responsible for managing system configuration (accounts and
sudo, networking, NTP, DNS, logging and log rotation, etc.) on
all Linux and Solaris systems.
- New client/server architecture with integrated controlled
deployment of changes, impact prediction for new changes, and
reporting.
- Design and implement a cross-platform application packaging system
- Released as Open Source:
http://tpkg.sourceforge.net/
- Responsible for managing the packaging and deployment of
customer-facing and internal applications
- Features include management of dependencies, rolling
deployments, and integrated hooks into system configuration for
accounts, cron, init, kernel tuning, etc.
- Design and deploy large-scale, private cloud computing and grid
computing environments for use by multiple teams within the
organization.
Yahoo!: April 2004 to April 2008
During my time at Yahoo! I progressed through several positions of
increasing responsibility.
December 2006 to April 2008,
Manager of System Administration,
Yahoo!
Manager of the 15+ person System Administration team. My team was
responsible for the deployment and maintenance of the Advertiser and
Publisher group's 15,000+ production systems which ran the Sponsored
Search (text) and Display (graphical) advertising applications and
generated a significant portion of Yahoo!'s revenue. My team included
employees in Burbank, California, New York City, Tokyo, Japan, Dublin,
Ireland and Bangalore, India who helped provide round-the-clock coverage
for our environment.
In addition to previous responsibilities:
- Lead the team during the Apex project to integrate the text and display
advertising businesses within Yahoo!, merging systems managed by three
different teams.
- Integrate the four member SA team from the Right Media
acquisition.
- Lead the team to develop proactive tools and processes for managing
our environment. A complete self-service user management system
allowing our customers to manage the accounts and sudo privileges for
hundreds of different applications while maintaining compliance with
the approval processes required by internal security teams and
Sarbanes-Oxley. Extensive hardware monitoring with automatic generation
of tickets to the on-site Data Center Operations teams for hardware
repairs. And numerous other tools and processes to ensure we catch
issues before they turn into problems.
May 2005 to December 2006,
Manager of System Administration Operations,
Yahoo! Search Marketing
Manager of the 8 person System Administration Operations team. My team
was responsible for the deployment and maintenance of the Search
Marketing group's 10,000+ production systems. My team included two
international employees (Tokyo, Japan and Dublin, Ireland) who helped
provide round-the-clock coverage for our environment.
- Led the team through the growth of the environment from 3,000 to
10,000+ servers with minimal staffing increases. At one point we
deployed more than 4,000 servers in one month.
- Led the team through the integration of the Overture/YSM and
Yahoo! environments, gradually bringing the YSM environment into
compliance with Yahoo! standards and practices.
- Established a strong working relationship between the YSM SA team
and three other SA teams within Yahoo!
- Led the team through the Panama project, a year-long complete
overhaul of both the systems and applications for the entire YSM
environment.
- Managed a 24x7x365 on-call schedule with multiple tiers and 15+
participants.
- Coordinated the interview and hiring process for multiple
employees. Wrote year-end reviews for team and made recommendations
for salary, bonus and equity compensation adjustments.
- Ensured team compliance with SLAs in handling our ticket queue.
The SA team consistently maintained one of the best rates of SLA
compliance within YSM. In conjunction with the YSM Reporting team I
pioneered the development and use of quarterly reports on individual
effectiveness in responding to ticket requests, which we used to
reward our top performers and mentor lower performers. I also
oversaw and monitored team compliance with YSM change management
practices.
- Coordinated team compliance with Sarbanes-Oxley requirements. Met
with internal and external auditors on a regular basis to discuss our
practices and procedures with regards to account provisioning and other
security practices.
April 2004 to May 2005, Senior System Administrator,
Yahoo! Search Marketing
(formerly Overture)
Member of the 15+ person System Administration team within Production
Operations, responsible for the operation of the 3000+ computer systems
in YSM's production operations. Manage a mixture of Red Hat Linux,
FreeBSD and Solaris systems providing hundreds of millions of
advertisements daily for YSM's partners, including Yahoo!, CNN and MSN.
- Progressed to the architecture team, and then to team lead
within 6 months of being hired.
- Led the project to upgrade all Linux systems to Red Hat Enterprise
Linux.
- Led the deployment of over 500 new servers.
- Led the deployment of the configuration file management system I
designed (see Accomplishments section).
University of Washington: July 2001 to April 2004
Senior Computer Specialist,
Electrical Engineering Department
at the University of Washington
System administrator for the Electrical Engineering department
at the University of Washington. Responsible for maintaining a mixture of
Linux and Solaris servers and workstations. Services provided include
mail (SMTP, S/IMAP, S/POP), NFS, Kerberos, LDAP, NIS, Samba, print services,
DNS, DHCP and HTTP/HTTPS.
- Deployed the configuration file management system I designed (see
Accomplishments section).
- Coordinated roll-out of Red Hat Enterprise Linux.
- Implemented Kerberos and LDAP services for
user authentication and Unix user information.
- Implemented a two-tier backup system for user data: The first
tier system uses rsync to copy data to a remote system and creates
daily snapshots. The second tier uses traditional tape archives.
This allows for data growth while reducing the need for expensive
new tape hardware.
Collective Technologies: June 1997 to July 2001
System Administration and Security consulting with
Collective Technologies.
I worked on numerous projects at a variety of clients, including
AT&T Wireless Services, Aventail.Net and Hughes Space and
Communications.
California Institute of Technology: May 1995 to June 1997
As Webmaster for the Campus Computing Organization at
the California Institute of Technology
I maintained the main Web pages for the California Institute of
Technology.
Education
- Bachelor of Science in Biology, California Institute of Technology
(Caltech)
- Introduction to Cisco Router Configuration, Advanced Cisco Router
Configuration courses
- USENIX LISA conference and tutorials in 1999, 2002, 2003, 2004,
2005, 2006, 2007 and 2008.
- USENIX BSDCon conference in 2000, including Marshall Kirk
McKusick's BSD Internals tutorial
- RubyConf 2009
- SCALE 2010
Accomplishments