KatysBlog

This text was originally published on 24 July 2009. It was modified on 14 August 2009 based on a re-analysis of the data prepared for the Sun Labs “Sun Mentoring: 1996-2009” Technical Report. The major change was to convert the “Comparing Boom, Between, and Bust” numbers from per-year to a 3-year analysis to show changes over time.

Sun Mentoring Programs

Sun has offered several internally-developed formal mentoring programs since 1992, three of which are still available:

• SEED, for which I have been the Director since 2001.
  SEED has four subgroups: Recent Hires, including New College Hires (2001), Established Staff (2002), PreSEED (2008), and special pilot terms for specific geographies, professional areas, or new acquisitions (2005-2009)
• Mentoring@Sun, managed by Helen Gracon since 1996
  This program includes both open enrollment and intact work groups
• New Sun Vice Presidents’ Mentoring , managed by Helen Gracon since 2004

Recently, Helen started reporting to me, so we are taking advantage of our pooled knowledge by creating this mentoring series. Tanya Jankot (SEED’s Applications Engineer), Helen Gracon, and I have been analyzing our mentoring program data, ably supported by our Sun Human Resources team, including Sy Dimitroff ,and Matt Artz . Because much of the data we are using is private and confidential, we are limited to what we can publish.

Mentoring and the Economy

One question we wanted to answer was: “Is mentoring success tied to the larger economy?” How much of the very positive metrics that we see
in mentoring programs for promotion, higher ratings, retention, satisfaction, etc. are because a rising tide lifting all boats? That is: does mentoring make a big difference in spite of general economic improvements benefiting all participants? Fortunately, we have a great deal of internal-to-Sun data we can analyze about our mentoring program participants. Also, Mentoring@Sun was the subject of a formal research study…

Gartner on Mentoring@Sun

The Mentoring@Sun study is:  “Case Study: Workforce Analytics at Sun” by James Holincheck, Gartner Research ID #G00142776, Publication Date: 27 October 2006.

For this study, Helen worked with Capital Analytics, pulling data from 1998-2001 about 95 mentor-mentee pairs who participated in the Mentoring@Sun program in 1999. (The data was collected for one year before the Mentoring@Sun terms and for two years after.) All were in one of four intact work groups.  Three of the groups studied were in Sun Engineering and one was in Sun’s Worldwide Operations. The 95 pairs were compared against almost 1,500 members of a control group of Sun staff (taken from the same business groups). The data were analyzed in 2002 and published by Gartner in 2006.

Tanya and Helen and I reviewed the 1998-2001 data and 2002 analysis, then checked back with Dr. W. Boyce Byerly (Chief Scientist and CTO of Capital Analytics, who worked on the original study) with questions for this report. We found some differences in operational definitions. For example, in Gartner Figure 3, the labels of Administrators (8.5% change in salary grade), and Engineers (6.2% in salary grade) are switched. The Sun group called “Administrators” in Gartner’s report were actually salaried (exempt, senior grade) Engineers, so their 8.5% change in pay grade makes more sense. Those called “Engineers” in the Gartner report were really Sun salaried non-technical staff. However, since the switched labels were used consistently, the numbers and analysis are still valid, but not the conclusions. Another operational definition which did not match Sun’s standard usage has to do with high performance. In the Gartner report, the “high performers” were those who had the highest salary before the beginning of the mentoring program. Sun uses
“high performer” or “high potential” to mean staff who routinely get Superior annual performance ratings. Again, the operational definition was consistent so we were able to compare the data.

Gartner’s positive findings were in the areas of change in salary, promotion, and retention. Gartner also had a negative finding:

“…investing in a mentoring program for high performers does not yield as significant a return as might be assumed. Rather, the better investment for Sun would be to spend the money on lower performers to help them raise their level of performance.”

This last finding is similar to the analysis of an excellent Harvard Business Review report called “Let’s Hear It for B Players” (by Thomas J. Delong, Vineeta Vijayaraghavan, Jun 01, 2003. Prod. #: R0306F-PDF-ENG). Because B players make up the great majority of employees: 80% of a company (as opposed to the top 10% of star A players, and the bottom 10% of incompetent C players), providing them with mentoring has a similarly larger benefit. This HBR article was one of the inspirations which lead to the creation of Sun’s popular PreSEED mentoring group in 2008, because, as Delong and Vijayaraghavan wrote:

“Like all prize-winning supporting actors, B players bring depth and stability to the companies they work for, slowly but surely improving both corporate performance and organizational resilience…. They will never garner the most revenue or the biggest clients, but they also will be less likely to embarrass the company or flunk out…. these players inevitably end up being the backbone of the organization.”

Other Sources

In addition to the external-to-Sun Gartner report, we used a Sun-internal report prepared by SEED’s former Program Manager Justin Yang. In “1996 – 2000 Engineering New College Hire Data Summary”, Justin Yang analyzed information from 485 new college hires with the title Member of the Technical Staff (MTS-1 through MTS-4 seniority levels). In 2002, I asked Justin to prepare this report so that we would have a baseline against which to compare future performance of the then-newly-created SEED mentoring program. For boom and bust cycle date ranges, I referred to Wikipedia articles such as:  “List of recessions in the United States” and “Dot-com bubble”. I also checked on Sun’s history using “The Motley Fool – Sun Microsystems, Inc. (JAVA)” and the Sun Microsystems – Annual Report Archive. The data in the Gartner report were pulled during the “dot-com bubble” of 1998-2001, as was most of the data in Justin Yang’s report. The information in these two reports was clearly collected during boom times. The worldwide recession (which started in 2007) represents a bust time for the Silicon Valley in general and for Sun Microsystems in particular.

Calculating ROI for Mentoring

Calculating Return on Investment (ROI) for mentoring is dependent on assumptions and variables used. In 2002, Capital Analytics used the following formula to calculate the return on $695/person paid to SunU for the 95 mentor-mentee pairs in the 1999 Mentoring@Sun program.

(Return – Cost) / Cost

Dr. W. Boyce Byerly confirmed that Capital Analytics found 1,000% ROI, for Sun mentoring, using their most conservative measures of job and salary grade improvement. Their analysis methods are published in the 2004 paper on ProCourse ROI software “Measuring the True Business Impact of Training”.

Mentoring@Sun is offered at a per-participant charge by SunU (the former name for Sun Learning Services). The SEED program is offered for free to participants (program costs were covered by the Chief Technologist’s Office). This difference in how the program costs were covered probably does not effect the ROI.

Some of the assumptions used in this ROI calculation may be controversial:

• Compensation paid to employees reflects their value to the company.
• A dollar increase in compensation reflects a dollar increase in value to the company.
• Higher compensation in the years after mentoring program participation is reflective of that participation.
• The company will recognize improvement in value, and increase compensation accordingly.

Triple Creek is a mentoring service company which was not involved in the 1998-2001 Sun case study but has published an interesting analysis using the well-known 2006 report by Gartner. In Triple Creek’s 2007 paper “Mentoring’s Impact on MENTORS / Doubling the ROI of Mentoring”, an ROI of 1,500% to 1,710% was calculated.

Analyzing Different Groups Over Ten Plus Years

Since there are many variables, what we present here is more a broad indication of patterns than a targeted scientific study. There are a variety of mentoring terms (or individual groups) represented:

• Some were Sun-wide terms but others were limited to Sun Engineering.
• Some terms were for senior or high-potential staff but others available to anyone who could get management approval (self selection).
• Some terms were created through open enrollment, others included intact workgroups, many were selected by competitive application.
• Most terms were sponsored by an executive.
  • Greg Papadopoulos (Sun’s Chief Technology Officer and Executive VP of Research and Development) sponsored over thirty terms.
  • Karen Rohde (Senior VP of Human Resources and Sun’s Chief Talent Officer) and
  • Bob Worrall (Senior VP of IT Operations and Chief Information Officer) each sponsored five terms.
• All of the staff who took the mentoring programs worked for Sun Microsystems as regular employees (not interns, contractors, or temporary staff) for at least some of the time from 1996-2009.

Sun’s mentoring programs are voluntary: the mentees and mentors may be encouraged to participate by their managers or peers but the programs are not remedial (not for people on a required performance improvement plan, for example). People join a mentoring program for different reasons. Three common reasons to join are:

• They are curious and want to learn.
• They are ambitious and motivated to improve their career.
• They are stuck personally or professionally and want to develop a new way to proceed.

Read the entry on Formal vs. Informal Mentoring to learn more about why a participant might choose one type of mentoring over another.  For some measures, we have more specifics than others, for example:

• Gender
  All the terms included mixed gender mentor-mentee pairs.— SEED has an average 20% female mentee participation, and 15% female mentor participation, 2001-2009. This reflects the lower percentage of women in Engineering than in Sun overall. SEED’s range is 0% to 30% women mentees per term. The Recent Hire and Established Staff SEED mentees had the highest percentage of women (22%) while the special pilot programs were much lower (17%). Since 2001, women and non-US staff have taken advantage of SEED at a consistently higher rate than their representation in Engineering
  overall.
  — Mentoring@Sun included Engineering and non-Engineering staff but gender data were not collected for all terms. The Mentoring@Sun
  range is 5% to 75% women mentees per term, reflecting the higher percentage of women in Sun overall than in just Engineering.
• Distance
  In most terms, the majority of pairs were working at a distance (in different cities, states, or countries) rather than local to each other.
  — SEED had 88% mentor-mentee pairs working at a distance, 2005-2008
  — Mentoring@Sun had about 75% mentor-mentee pairs working at a distance, 2005-2008
• Satisfaction
  We do not have complete metrics in all time ranges for all three mentoring programs. What we have:
  — SEED has quarterly satisfaction ratings from 775 mentees averaging 90% (2004-2008). In addition, 93% of SEED mentees reported that meetings with their mentor were worthwhile. 83% of mentors believe their Mentee’s participation in the SEED program made them more valuable to Sun (from 330 mentor reports). 88% of mentors said they wanted to be a SEED mentor again.
  — In the New VPs program, almost all participants rated program as effective or very effective and agreed to mentor a new VP by the end of the program. Almost all mentors and mentees report recommending the New VP Program to their peers.
  — We do not have consistent satisfaction measures for the largest of the three programs, Mentoring@Sun, but the reports we do have are
  very enthusiastic.

Sun’s mentoring programs are different in numbers of mentors and mentees:

• Mentoring@Sun: about 6,000 mentees and 4,500 mentors (1996-2009)
• New Vice Presidents: 138 mentees and 87 mentors (2004-2009)
• SEED: 1,162 mentees and 474 mentors (2001-2009)

There is overlap and duplication between the mentors in the three programs (these are very generous people!). Also, about 25% of current SEED mentors were originally SEED mentees. The totals for these three mentoring programs are about 7,300 mentees and 5,000 mentors.

We decided to focus on three measures for which we have the most information:

• Attrition (higher voluntary termination, opposite of retention, lower numbers are better)
• Compensation (salary increases, pay raises, higher numbers are better)
• Promotion (increase in job seniority or salary grade, higher numbers are better)

In context, these three metrics can be compared between the various sets of mentoring program information without being distorting or misleading. “Context” includes understanding larger population patterns than just those in the area of research:

• These numbers may or may not be representative of overall patterns. For example: because we do not know exactly how many new college graduates Sun hired 1996-2000, we cannot say what percentage of that population is represented by the 485 Members of the Technical staff in Justin Yang’s report. However, we do have some contextual glimpses. In the year 2000, we estimate there were over 500 new college graduates hired in all of Sun. So, for 2000, Justin Yang’s report covers roughly 1/5 of the population of all new college graduates hired.
• The three metrics do not stand alone; they interact with each other and other measures and are tied to many factors having little to do with mentoring.
• New College Hires (such as those in Justin Yang’s report) seem to be a special case. For example: his report showed that there is a higher retention rate for more recently hired staff. Promotion is tied to retention: if New College Hires are promoted, they are more likely to stay. The more recent hires were promoted more quickly.

Comparing Boom to Bust

We used the three measures of Attrition, Compensation, and Promotion during three time periods:

• Boom (1998-2001)
• Between (2002-2006)
• Bust (2007-2009)

Based on the results shown in the table below, the following conclusions can be drawn about the performance of participants in Sun’s mentoring programs:

1. Attrition went down after the Boom period and then went down again during the Bust.
2. Pay raises (Compensation) went up substantially after the Boom period, and continued high during the Between and Bust periods. However, raises fell slightly during the Bust (although Bust period raises were still higher than during the Boom period).
3. Promotions went up substantially after the Boom period. Promotions fell by 38% during the Bust period but were still much higher than during the Boom period.

Circumstances which may help in understanding these conclusions:

1. SEED mentoring program performance numbers may show more success because the program is focused on selecting high potential future engineering leaders, who are then given additional support to help them succeed. The success of the individual participants is due to their own capabilities and hard work (plus available opportunities and good management!). Increased success of the participants as a group may be attributable in part to the SEED program.
2. The 1,082 SEED mentees included senior and junior Engineering staff. However, when Tanya Jankot ran the numbers for just the junior staff (Recent Hires and PreSEEDs) in SEED, the results were only slightly different than for overall SEED performance.
3. The four Mentoring@Sun groups in the 2002 Capital Analytics study were intact work groups, three from Engineering and one from Worldwide Operations. The Capital Analytics control group was taken from the same work areas.
4. As described above, Engineering New College Hires seem to be a special case, especially in terms of their promotion and retention patterns;
   however, since SEED includes a Recent Hire group which includes some New College Hires, their patterns are important.

The question we wanted to answer was: “Is mentoring success tied to the larger economy?” Based on these analyses, in the case of the Sun
mentoring programs, it seems that success is only loosely tied to the performance of the larger economy. The Bust period caused both Compensation and Promotion numbers to fall but both remained substantially higher than during the Boom period. Participants in Sun’s mentoring programs outperformed control groups and participants show remarkable success in all measures.

.	Boom (1998-2001)	Between (2002-2006)	Bust (2007-2009)
Attrition	GAR-mentoring: 28% attritionGAR-control: 51% attritionECH: 26% attrition	SEED-rolling: 20% attrition	SEED-rolling: 14.3% attrition
Compensation	CA-mentoring: 7.8% average base salary increaseCA-control: 4.2% average base salary increase	SEED-rolling: 15.8% average base salary increase	SEED-rolling: 13.2% average base salary increase
Promotion	GAR-mentoring: 25% promotedGAR-control: 5.3% promotedECH: 47% promoted	SEED-rolling: 65.6% promoted	SEED-rolling: 40.3% promoted

Reference Key:	.
CA–mentoring	2002 Analysis by Capital Analytics of 1998-2001 data on 95 mentees, in four Mentoring@Sun groups. CA-mentoring is compared to CA-control. 3 year study.
CA–control	2002 Analysis by Capital Analytics of 1998-2001 data on about 1,500 Sun staff in a control group (not in a mentoring program). CA-control is compared to CA-mentoring. 3 year study.
ECH	“1996 – 2000 Engineering New College Hire Data Summary” – 1996-2001 baseline data on 485 junior Sun Engineering staff recently hired out of college (not in a mentoring program). ECH does not have a control group. Data shown is last 3 years of a 5 year study.
GAR–mentoring	Gartner “Case Study: Workforce Analytics at Sun” – based on Capital Analytics’ 1998-2001 analysis on 95 mentees, in four Mentoring@Sun groups. GAR-mentoring is compared to GAR-control. 3 year study.
GAR–control	Gartner “Case Study: Workforce Analytics at Sun” – based on Capital Analytics’ 1998-2001 analysis of about 1,500 Sun staff in a control group (not in a mentoring program). GAR-control is compared to GAR-mentoring. 3 year study.
SEED–rolling	Sun Engineering-wide world-wide mentoring program data on 756 mentees (2001-2007). SEED does not have a control group. SEED changes over time are compared with SEED itself for this analysis. In this table, the rate of population attrition, promotion, and salary increase are over a three year period (during and 2-years post-SEED-participation) and are calculated as an average over the population of mentees who participated in a mentoring program during the given years included in the Between or Bust cycle.

Series

Other entries in this series on mentoring are in the 2009 Sun Labs “Sun Mentoring: 1996-2009” Technical Report.

For more about SEED, the Sun Engineering 2001-2010 worldwide mentoring program, see SEED’s Facebook home page.

19 May 2016 – Links were updated