Tag Archives: Sun Microsystems

Mentoring in Engineering and Computer Science

Work to balance the diversity of Participants in terms of demographics, professional area, and geographic location.

The context of the Engineering community is key here. It bounds the SEED program and defines its organizational character. Compare this to the Mentoring@Sun program, started in 1992 as a general Sun-wide mentoring program. The SEED program was developed by Sun in 2001 to address Engineering organization needs that were not met by Mentoring@Sun. That is, SEED is an internal mentoring and leadership growth program designed to meet the needs of a key professional area, running in parallel with a more general internal program.

  • Both SEED and Mentoring@Sun are very effective at making connections between organizational silos, what Helen calls cross-pollinating.
    For more on this, read my blog entry Internal or External Mentoring Program? (30 June 2009).Scope:
    SEED mentors can be from any part of Sun so long as they are at principal-level or above in seniority. SEED mentees, however, must all be working in Engineering, which is defined as:

    “Hardware and software engineering positions where the primary job purpose is to perform engineering research, design, and development activities resulting in innovative Sun products for external customers. Also included are staff positions providing strategic support to engineering research, design, and development activities.”

    Again, the Engineering professional context provides specific program boundaries: only these positions are included, others are not. (This would sound like inappropriate exclusivity if Engineering did not make up about half of Sun’s employees.)

    Training Focus:
    Each mentoring program should provide training that helps the pairs feel comfortable from the start and work well together for the entire term. Training is particularly important in special cases, such as when mentor and mentee work in different professional areas (Microelectronics and Finance, for example), have a wide gap in their relative experience or seniority (such as a Senior Director mentoring a recent college hire Member of the Technical Staff), are working at a distance (for over half of SEED mentoring pairs, the mentor and mentee work in different cities, states, or countries), or come from very different cultures.SEED offers two hours of individual training by phone for each mentoring pair. Using a standard set of materials (Helen and I update these annually), pair training is tailored to their strengths and challenge areas. The geek personality is common enough that our mentoring training materials have a special section for Engineering. Engineers are professional problem solvers who are usually very smart analytical logical thinkers. Sometimes it can be a stretch for them to see the other person’s point of view. Many of them do not suffer fools. Mentoring training for extreme geeks may focus on teaching how to disagree agreeably (using tactful phrases) and learning when problem solving may not be what is needed or wanted by their mentoring partner.

    Management Style:
    Managing an Engineering mentoring program requires communicating well and maintaining trust with Engineers. SEED is a prestigious leadership grooming program, so the decision of which applicants get accepted can be controversial. The selection system must be fair and seen to be fair. Selection criteria for SEED are based on the values of the Engineering community (such as: demonstrated technical excellence, creativity, leadership, holding patents, publishing papers, earning an excellent letter of recommendation by an executive, etc.) Many of SEED’s selection criteria are also reflected in job promotion criteria for Engineering staff. Sun Engineering has an egalitarian open door culture which values data-driven decisions and a transparent management style. While respecting confidentiality, SEED routinely makes a great deal of program information available to Sun Engineering. SEED program participants regularly contribute suggestions on how to improve the program and its web tools.

    What is the Geek Personality?

    A brief digression into the personal and social context of Engineers since this has such a strong influence on mentoring in Engineering…

    While Sun Engineering staff include a very broad range of personality types, there are some unusual concentrations. SEED mentoring training includes a section on Myers-Briggs style personality types. This provides a good context and vocabulary for mentoring pairs to discuss differences and commonalities and promote mutual understanding. (We skip this section of training for staff who think the use of personality types is Psychology black magic.) Sun used to offer personality assessments as part of its regular career coaching benefit. In 2002, I used a survey to collect information from 143 Sun Engineering staff about their formally assessed personality type. While not a statistically valid sample, it is nonetheless interesting:

  • 59% of the Sun Engineering staff reported that they had been assessed as I
    (introvert)About 50% of the US population are I (introvert)
  • 66% of the Sun Engineering staff reported that they were NT (intuitive thinkers)About 10% of the US population are NT (intuitive thinkers)

(Yes, this does mean that Engineers are abnormal, statistically at least).

Introverts have been defined as “people who find other people tiring” (see “Caring for Your Introvert” by Jonathan Rauch, The Atlantic, March 2003). A t-shirt popular with Engineers says “You read my t-shirt. That’s enough social interaction for one day.” (see Think Geek T-shirt). SEED works hard to make its communications comfortable for an introverted group. For example, we lay out the expected interactions and always allow the participants to engage at their own comfort level. One analysis of downside to being an introvert is that:

      “In our extrovertist society, being outgoing is considered normal and therefore desirable, a mark of happiness, confidence, leadership. Extroverts are seen as bighearted, vibrant, warm, empathic. ‘People person’ is a compliment. Introverts are described with words like ‘guarded,’ ‘loner,’ ‘reserved,’ ‘taciturn,’ ‘self-contained, private’ – narrow, ungenerous words, words that suggest emotional parsimony and smallness of personality. Female introverts, I suspect, must suffer especially. In certain circles, particularly in the Midwest, a man can still sometimes get away with being what they used to call a strong and silent type; introverted women, lacking that alternative, are even more

 

    likely than men to be perceived as timid, withdrawn, haughty.”(Ibid, 2003 article by Jonathan Rauch)

For more on Social Context, Gender, and Mentoring, see my blog entry Picking Your Mentor, Picking Your Mentee.

Finding Mentors for Engineering

Since 2001, I have matched almost 1,200 mentoring pairs; 70% of the mentors were executives (Directors, Vice Presidents, Principal Engineers, Fellows, etc.). SEED gets an average of 90% participant satisfaction rating on its quarterly reports, year after year. What do these executive mentors look for in mentees? Why do so many find SEED to be such a satisfying program? Most of the questions mentors ask when I contact them about working with a potential mentee are structural: availability, time commitment required to participate, potential areas of difficulty (like being in the same management chain or speaking different primary languages), and physical or time zone proximity are common questions. Along with those are asked more substantive questions about intellectual common ground, interests, and personal compatibility. Somewhere in this mix, almost all potential mentors ask something like “Why me? What does this person want to know that I am uniquely able to teach?” (For more on mentor questions and preferences, read my 6 July 2009 blog entry Picking Your Mentor, Picking Your Mentee).

Notice that relatively few questions are about the topic or professional area to be discussed. SEED Mentors have served from all areas of Engineering worldwide, plus Operations, Sales, Service, Legal, Information Technology, Finance, Human Resources, and Marketing. Most of the non-Engineering staff were recruited as SEED mentors at the specific request of a mentee who asked to learn from them. I originally recruited the General Counsel as a mentor because a Software Engineer wanted to learn more from the lawyer’s success as a business leader. (He enjoyed the experience and has served as a mentor five times since.) I recruited a Finance Vice President because a Systems Program Manager wanted a mentor who really understood financial planning, revenue and cost management. Sun Microsystems is an Engineering-driven company, so most non-Engineering staff are eager to help (as well as extend their own connections in Engineering).

I have observed that the more experienced or senior a mentor is, the more willing they are to discuss a very broad range of topics. It is usually the more junior mentors who question their breadth of ability or the value of their experience outside of their immediate area of professional expertise. The mentors who seem to get the most out of their SEED experience are the executives. One Software Vice President told me that his hour with his mentee was his vacation, the only time all week when he knew the answers. A different Software Vice President told Helen that he always looked forward to meeting with his mentee: it was his only non-confrontational meeting. This positive experience is reflected in SEED’s metrics for repeat mentor participation:

  • 48% of the total 460+ potential mentors on SEED’s current list have been mentors more than once. This includes principal-level senior staff
    plus executives.(This does not count their service in Mentoring@Sun or other Sun mentoring programs.)
  • 65% of those repeat mentors are executives.
  • 54% of all of the executives who have ever been SEED mentors have mentored more than once.45 executives have have served as a SEED mentor five or more times.4 Sun executives have mentored ten or more times with SEED.

A Marketing Vice President wrote in evaluation of his sixth SEED mentoring experience:

“This continues to be a great program and I get a lot out of it — possibly more than the mentees.”

Series

Information is from my experience since 2001 managing Sun’s SEED Engineering-wide world-wide mentoring program, and from the Mentoring@Sun general mentoring program, and the mentoring program for new Sun Vice Presidents managed by Helen Gracon since 1996. Helen Gracon also provides training for SEED. This is part of a continuing series on mentoring programs. Other entries in this series:

For more about SEED, see the program home page at http://research.sun.com/SEED.

By Katy Dickinson
Director, Business Process Architecture
Chief Technologist’s Office & Sun Labs, Sun Microsystems

1 Comment

Filed under Mentoring & Other Business

Mentoring Program Web Tools and Process

Web tool design is a technical art which requires an unusual combination of software programming, usability engineering, and program management skills. As Director of Sun Microsystems’ SEED Engineering-wide worldwide mentoring program since 2001 (and the program’s Process Architect), I have been gifted with two talented staff members who can do this work. Tanya Jankot has been SEED’s Applications Engineer since 2003. Before Tanya  Justin Yang held the position of SEED Program Manager for two years.

SEED developed its own set of tools for mentoring program and information management. These tools have not been “productized”. Why not use an external-to-Sun set of web tools? For the answer, read my 30 June 2009 blog Internal or External Mentoring Program?

Tools and Process Overview

The original SEED program was based on a year-long need analysis and program design in 2000 by a team mostly made up of Sun Human Resources (HR) and Engineering staff. The process itself was created on-the-fly during the first pilot term in 2001. The SEED mentoring program has expanded to a much larger audience in recent years and SEED’s web tools have developed and been redesigned accordingly. In the program’s first year, 2001-2002, there was just one term. In 2008-2009, there have been 12 overlapping terms in four groups (Recent Hires, Established Staff, PreSEED, and special pilots). Since the SEED team and I were creating a new mentoring system essentially from scratch in the 2001 pilot term, our guidelines for process and tool development were:

  1. Keep it simple
  2. Check in with customers and stakeholders frequently
  3. Only include the minimum: question the need for each step before it goes in, and again at every review, and again before publication
  4. Let the process define the web tools
  5. Assume that process and tool users will have access to only the most basic web resources and performance
  6. Collect and analyze data routinely and make decisions based on those data

These guidelines have continued to serve SEED well. We also kept using the concept of a “pilot” to expand the program. In pilot terms, the rules, process, and/or scope are somewhat different from the regular SEED program. The Established Staff group was created in 2002 and the PreSEED program was created in 2008 using pilot programs; both have been very popular offerings. A pilot allows us to put something imperfect out there to see
what works. Sometimes pilots fail (for example, the SEED-2 or SEED Alumni term in 2007 only attracted ten participants).

SEED now has two major formal processes, for participant selection, and for mentor selection. These processes are published in full detail for the use of Sun internal program participants. Flow charts are also available in the appendix of “Sun Mentoring: 1996-2009” the Sun Labs 2009 Technical Report  (includes a copy of the Research Disclosure Database Number 482013, defensive publication from Research Disclosure, Published in June 2004, Electronic Publication Date : 17 May 2004).

In November 2008, Tanya created and gave an internal-to-Sun presentation on developing simple web technologies using the SEED tools as examples. Her presentation was created to educate other Sun project teams and web teams. The information following about SEED web tools is derived from Tanya’s presentation. Her overview statements about the SEED’s current web tools:

  • The technology was built to model SEED mentoring processes which were already designed and pilot tested (we tried to fit the tools into the existing work flow rather than build processes around the tools).
  • The tools have evolved with the program: need for greater automation to allow scaling, new requirements as the program expanded across geographical areas regions, organizations, etc.
  • SEED relies on existing Sun corporate data systems as much as possible, only
    gathering additional information not already available elsewhere.

More specific details follow about SEED’s web tools and the technology and process behind them. Screen shots and other confidential data have been removed from Tanya’s original presentation material.


Developing Simple Web Technologies for the SEED Mentoring Program

Tools Behind the Program

  • The systems supporting the SEED program have evolved with it over time. They currently include:
    – A system to manage each term’s application process. Details are in Term Application Materials and Term Application Management, below.
    – A system to support the mentor matching process. Details are in Mentor Request Management, below.
    – An archive of program applicants, participants, mentors, etc. which enables long-term program management and metrics. Details are in
    SEED Program Database, below.
    – Applications to support regular program activities, such as regular quarterly feedback reports and bi-annual events for mentees, mentors, and the mentee’s managers.
  • Tool development goals:
    – Ensure the integrity and confidentiality of applicant and participant data.
    – Increase the ease-of-use for program participants and SEED staff.
    – Increase program efficiency and quality of data available to the SEED team, extend the number of participants, raise the value of participant experience, and justify their trust in the program

Simple Technologies

  • “SAMP” (Solaris, Apache, MySQL, and Perl & PHP)
  • htaccess and Sun confidential employee records access authentication
  • Queries to the Sun confidential employee records system
  • Email
  • Other technologies available within Sun and Sun Labs, such as a name auto-suggest widget and a survey tool.
  • And still making use of old-fashioned static web pages

Term Application Materials

  • SEED terms have an application period, usually lasting two to three weeks, with firm deadlines.
  • htaccess and Sun confidential employee records access are used for authentication
  • In addition to submitting a completed application form, applicants must also submit their resume, their manager must submit a letter of recommendation, and in some cases they must also secure additional letters of recommendation from Sun executives. All materials are submitted through web-based forms.
  • Design considerations
    – Applicants are located worldwide.
    — Application materials need to be as clear and simple as possible because for many employees, English is not their primary language.
    — Applications must be functional on all Sun systems and locations. This includes Sun hardware with Solaris software, Sun Ray systems, experimental systems, as well as a variety of Macs, laptops, and PCs.
    – Application materials must reflect Sun’s organizational structure and HR policies in an understandable way. Many applicants are new to Sun and are not familiar with its organization or policies.
    — Divisions, organizations
    — Job Codes, titles
    – Manager and executive recommendation letters are submitted confidentially but are a required part of an application. A secure mechanism is needed for applicants to view the status of their application but not the details of all materials.
    – In order to ensure that all materials are submitted correctly and not “lost” (i.e., a recommendation letter is submitted against an incorrect applicant SunID), Sun confidential employee records system lookup, email confirmations, and SQL audits of the database are used.
    – Each term’s application materials are stored in a separate database for easy management. Key applicant data that needs to be tracked long-term is loaded into the SEED archive database at the end of the application period.

Term Application Management (SEED Team Website)

  • A central website used by the SEED program staff to efficiently track the status of applicants and their materials.
  • At the end of the term application period, key applicant data is verified against Sun’s Human Resources records.
  • Used by SEED’s executive selection committee to review each applicant thoroughly.
  • Tool goals
    – Present useful summary data in a small amount of space.
    – Accurately reflect the status and materials received for each applicant.
    – Allow a complete review of each applicant’s submitted materials.
    – Make information easy to find to answer questions from applicants, participants, managers, and mentors quickly and accurately.
    – Print in a useable format.

Mentor Request Management

  • Upon acceptance to the program, all participants are required to submit a 10-name “Mentor Wish List” of mentors they would like to work with.
  • At the close of the mentor request period and receipt of all wish lists, the SEED program staff begins the mentor match process. For each participant, the goal is to match them with the highest priority eligible mentor from their Mentor Wish List.
  • A decision is made in each case where more than one Participant requests the same potential mentor. In SEED’s current terms, 80 mentees prepared 10-name lists, which resulted in 387 unique mentor requests. There were 10 potential mentors with multiple 1st Priority requests and 39 mentors who were requested by 5 or more mentees. This is a common problem: as many as twenty-two(!) potential mentees in one term have requested the same mentor. The primary basis for this decision is the priority order on the Mentor Wish List provided by the Participant. The Participant’s seniority (number of years at Sun) may be used as a tiebreaker, with the more senior Participant getting preference.
  • Tool Requirements
    – For each mentor requested, both name and SunID are required to be entered due to variations in name entry and frequent errors in entering SunIDs.
    – A name auto-suggest widget has been very useful in creating cleaner submissions.
    – The Mentor Request form includes a validation step in order to check for known conflicts in the SEED Potential Mentors list.

SEED Program Database

  • The SEED program database drives the long-term management of the program.
  • It allows us to track past applicants, participants, and mentors, as well as manage our list of 450+ Potential Mentors: mentors who have volunteered to work with program participants.
  • These records allow for regular metrics analysis of the program, currently done annually. Automated metrics tracking is a goal that is in progress.
  • Challenges
    – Maintaining the data so it is meaningful over time. For example, divisional organizational changes (reorg) make it difficult to summarize the number of participants we have had from each organization over the life of the program.
    – Keeping the mentor records up-to-date: removing broken links, updating titles in a timely way.
    – We need to maintain records of all program mentors, participants, and applicants, even after they have left Sun.

Conclusions

  • The systems and tools that support SEED have evolved with the program over time. They are not a single unified system, but being modular are easy to modify or extend when changes are required.
  • Using the technologies that are available and used by others allows you to be more efficient: learning from their work, and sharing components when possible.

Series

This is part of a continuing series on mentoring programs. Information is to answer frequently asked questions, based on my experience since 2001 managing Sun’s SEED Engineering-wide world-wide mentoring program. Other entries in this series were integrated into “Sun Mentoring: 1996-2009” the Sun Labs 2009 Technical Report.

25 October 2013 – links and text updated

Leave a comment

Filed under Mentoring & Other Business

What I Love About Sun – After 25 Years

I just filled in the order for my anniversary gift from Sun Microsystems. It feels odd to have worked at one company for 25 years (since June 1984). I have been thinking about why I have stayed so long. Here is what I love about Sun:

 

    • Smart people who create amazing technology.At Sun, I have had the privilege to work with many great innovators and those who foster innovation, including:
      Carol Bartz,
      Whit Diffie,
      James Gosling,
      Vinod Khosla,
      Greg Papadopoulos,
      Radia Perlman,
      Eric Schmidt
      (who hired me), and
      Ivan Sutherland.

      I believe that the world is a better place because of Sun Engineering.

    • Generous people who think beyond themselves.Sun is a company for people with big hearts.

      I am proud of both large and small gifts, particularly:

       

        • The 545 senior staff and executive mentors in my
          SEED worldwide Engineering mentoring program who spend time they rarely have to help others down the path they have walked
        • The anonymous donor of the Free Milk Foundation half gallon which quietly appears in our hall’s refrigerator every Monday morning
        • Sun’s CTO, Greg Papadopoulos, who has supported the
          Anita Borg Institute and its Grace Hopper Celebration of Women in Computing, the American Heart Association’s
          Heart Walk, the SETI Institute, and so many other worthy efforts
        • Sun’s commitment to ecologically responsible products, and to volunteering in our local communities worldwide.
    • People with a sense of fun.Sun Engineers will put unreasonable effort into creating a great joke.
      Ever since I watched the senior Software developers putting Eric’s office into the pond in 1985, I have enjoyed Sun’s pranks and April Fools jokes. Sun has a sense of humor.

As the Oracle-Sun proposed acquisition progresses, I hope that what I love about Sun will be valued, grow, and continue to create great technology with a passion which makes the world better.

7 Comments

Filed under News & Reviews

Remote Inauguration

We were very excited that Jessica, my 20-year-old daughter, attended this week’s Presidential inauguration in Washington, DC. You can see her blog entries and photos at  http://feelingelephants.wordpress.com/. Ours is a politically passionate family. One of my earliest memories was glee that my candidate (John F. Kennedy) won the presidential election over my older brother’s candidate (Barry Goldwater), in 1964.

Our family has always been split between liberal and conservative. The divergence of our current family politics is best shown in two objects: a framed picture of the late President Ronald Reagan that my father put up in the front hall of their San Francisco house (intended to be seen by everyone who came over for parties to phone Obama voters, hosted by my mother), and the shoe with BUSH –> in gold paint on the toe that someone gave my father for Christmas:

Bush Shoe photo: copyright 2009 Katy Dickinson

I was at work at Sun during Tuesday’s inaugural morning so I went over to the Menlo Park campus Crossroads conference room to watch history unfolding
live by CNN TV broadcast on the big screen. Because I usually get my news from National Public Radio (NPR), it was particularly interesting to see how the great and powerful look and interact:

    • Why did Jimmy and Roselyn Carter greet  George and Barbara Bush with a kiss for Barbara but then walk by  Bill and Hilary Clinton, who were right next to the Bushes, without a word?
    • The senior President Bush does not seem to be aging well. He sat next to Hilary Clinton and behind the new First Lady  Michelle Obama, so there were many pictures of him with his mouth open looking confused.
    • Hilary Clinton, on the other hand, looked radiant two days before her confirmation as our new Secretary of State.
    • It was fascinating to watch outgoing President George W. Bush during his last minutes in office. I saw Bush pat the leg of one of the tall Marines in full dress uniform as he walked past – like you would pat a friendly dog.

It was certainly a great day for San Francisco, with soon-to-be President Barack Obama walking into the ceremony right behind Senator Dianne Feinstein
and Speaker of the House Nancy Pelosi. This is a welcome change from President G.W. Bush pretending that California did not exist. Having Dianne Feinstein serve as Chair of the Joint Congressional Committee on Inaugural Ceremonies meant that we got to see a great deal of her between speakers and events. We are so proud of her!

Except when greeting people, President Obama seemed grim during much of the event. The only time I caught a big smile was when he messed up his inaugural
oath (he had to take it again later). The biggest smile of the day, however, was that of cellist  Yo-Yo Ma who appeared delighted to be performing with violinist
Itzhak Perlman. There was much wondering how the instruments and musicians could play “Air and Simple Gifts” so well on that cold day. This was cleared up when it was  announced today that those on the inaugural stage heard the musicians live but a prior recording was broadcast for everyone else. However
real the broadcast, Ma’s smile and the superb music were a genuine delight.

CNN in Sun’s conference room

Inauguration on CNN in Sun's conference room photo: copyright 2009 Katy Dickinson

Feinstein, Obama, Pelosi on CNN

Feinstein, Obama, Pelosi, inauguration CNN photo: copyright 2009 Katy Dickinson

Obama taking the oath, CNN

Obama taking the oath, inauguration CNN photo: copyright 2009 Katy Dickinson

Clintons, Obama, G.H.W. Bush, CNN

Clintons, Michelle Obama, G.H.W. Bush, inauguration CNN photo: copyright 2009 Katy Dickinson

Hilary Clinton, CNN

Hilary Clinton, inauguration CNN photo: copyright 2009 Katy Dickinson

Perlman, Montero, Ma, McGill, CNN

Itzhak Perlman, Gabriela Montero, Yo-Yo Ma, Anthony McGill, CNN photo: copyright 2009 Katy Dickinson

Photos Copyright 2009 by Katy Dickinson

1 Comment

Filed under News & Reviews, Politics

Sun Family Changes

My husband, John Plocher, was laid off yesterday from Sun. We will be packing up his office on the 3rd floor of Menlo Park 17 this morning if you want to come by and say goodbye to him. You can read John’s farewell blog and other information. Anyone looking to hire a very experienced senior software and systems architect and developer with solid experience in 6 Sigma program management and open source development process design, please contact John. He is smart, personable, and has a wicked sense of humor. This is an entirely objective analysis, of course!

I am still working at Sun – business as usual for all of my programs.

John Plocher 

John Plocher 2008<br /> photo: copyright 2008 Katy Dickinson

John teaching Arduino assembly 

John Plocher teaching Arduino assembly<br /> photo: copyright 2008 Katy Dickinson

John teaching Arduino assembly 

John Plocher teaching Arduino assembly<br /> photo: copyright 2008 Katy Dickinson

Images Copyright 2008 Katy Dickinson

1 Comment

Filed under Home & Family, News & Reviews

Sun’s Technology Advisory Board (TAB)

I usually write about my work here at Sun for the SEED worldwide Engineering mentoring program. Another of my programs is Sun’s Technology Advisory Board or TAB, which I manage for Greg Papadopoulos (Sun’s Chief Technology Officer and Executive Vice President of Research and Development).

What does TAB Do?
Since 2005, TAB has met several times a year to discuss key technical topics and trends, partnering to shape Sun’s technological vision for future development and product plans. The role of a TAB member is to support Sun’s Chief Technologist’s Organization in making recommendations to the CTO and Chief Executive Officer, as well as to the Chairman of the Board and Sun’s Board of Directors. Here they are at yesterday’s meeting:

TAB, Sun Microsystems Technology Advisory Board, Greg Papadopoulos, Steve Ward, Ivan Sutherland, Danny Hillis, Dave Patterson, Mike Splain, June 2008

TAB in June 2008
L to R Standing: Greg Papadopoulos, Steve Ward
L to R Seated: Ivan Sutherland, Danny Hillis, Dave Patterson, Mike Splain
Who are They?

Image Copyright 2008 by Katy Dickinson
Links updated 11 January 2017

1 Comment

Filed under Mentoring & Other Business

How to Survey, Part 2 (Best Practices)

In my 16 April blog entry How to Survey, I presented 3 sections: Key Questions, Tools and Services, and Reading. In this entry, I present some Best Practices based on my experience and the advice of two wise and capable women with whom I had the honor to work: Dr. Robin Jeffries and Dr. Kornelija Zgonc. All errors may be attributed to my misunderstanding, not their teaching!

The most recent survey completed by my department here in the Chief Technologist’s Organization at Sun Microsystems was the SEED mentoring program quarterly report for April 2008. See Mentoring Success Metrics (April 30, 2008) for details. SEED (Sun Engineering Enrichment and Development) has been collecting quarterly feedback from a web-based survey since 2002, so this is a mature example of a cyclic survey. The SEED survey is not anonymous. Most of the practices below are also appropriate for one-time surveys and for anonymous surveys.

Characteristics of a Good Web-based Survey (with examples from SEED):

  • It is Short. The SEED survey consists of 14 questions. One way to shorten surveys: don’t ask for information that can easily be mined from another source.
  • It is Easy to Use and Understand. Use pull down menus wherever possible to provide clear options. When a range of answers is possible, offer the same one-to-seven range, with “1” being low, “4” neutral, and “7” being high. State questions as simply as possible and test for clarity (if it is possible to misunderstand, someone will). Avoid jargon, abbreviations, and local slang.
  • It is Easy to Analyze the Responses. Use very few open text fields. Use a seven point range so that there is a clear low, neutral, and high (more on this below). “Does Not Apply” and “No Response” are always options. “No Response” is the default option (that is, the respondent must make an active change to answer).
  • For Cyclic Surveys – Prior and Future Versions are Comparable. Questions do not change much over time.
  • It is Trustworthy. Send a survey copy immediately in email to the respondent. Make survey analysis results available to respondents promptly. Actively protect private and anonymous information. Say in the survey introduction what will happen with the results (then, do what you say). Remember Robin Jeffries’ First Law of Surveys: “Don’t ask questions unless you are prepared to act on the results!”

The following Attributes of Poor Surveys list is material developed by Kornelija Zgonc, former Sun Chief Master Black Belt, and my Six Sigma mentor:

What’s Wrong?

  • Survey goals unclear
  • No forethought about your processes
  • Lots of yes/no questions
  • Lots of written questions
  • Focus on symptoms
Why it’s a Problem:

  • Take-aways unclear
  • Don’t know how to implement changes
  • Limited analytics; need big sample sizes
  • Unclear or unfocused questions
  • Get more questions, not answers!

The following Attributes of Great Surveys is also material developed by Kornelija Zgonc:

  • Goals, processes, and possible cause/effect relationships are analyzed up front
  • Widely-scaled numerical questions allow lots of analytics and keep sample sizes low
  • Only need a few written questions to address unforeseen situations or problems
  • Survey has action-oriented focus to generate solutions, not more questions

Why a 1 to 7 Range?
Multiple choice options make it easier to statistically analyze survey results. One of the common and energetic “discussions” among those who design surveys is what range to allow for numerical questions. Simply put: how many number choices should the respondent be offered? Too short a range (like: 1=bad, 2=neutral, 3=good) may not reflect an accurate subtlety of opinion. However, too many options can give a false confidence in the value and gradation of the answer. Don’t ask for more precision than your users are likely to know!

A range of seven is the best choice. When seven or more numbers are offered in a scale (like: 1=strongly disagree, 2=disagree, 3=disagree, 4=neutral, 5=agree, 6=agree, 7=strongly agree), the data collected behaves and can be analyzed like continuous variables. (Data are discrete if there are a limited number of values possible. Example: number of legs on a cat, number of letter grades possible on a test. Data are continuous when the measurements can have any value. Examples: time, weight.) This allows tremendous analysis flexibility because there are many more statistical tools for continuous data analysis than for discrete  data analysis.

Why Statistics Don’t Matter (sometimes) With all deference to my colleagues who are statisticians and Six Sigma Master Black Belts, sometimes statistics don’t matter.

  • The survey itself is a form of communication, regardless of whether it is answered, analyzed, or acted on. The survey may change the nature of the audience’s awareness.
  • If you don’t ask the right audience or collect enough responses, the answer does not matter.
  • Some people will never give a top or bottom score under any circumstances.
  • Refine, reduce, remove:
    • Too many surveys make people hate or ignore you.
    • Too many questions will cause your audience to abandon the survey part way through.
  • If your questions are too personal or respondents are embarrassed to tell the truth
    (for example: admitting they don’t know the answer), answers will be worthless.

Links and formatting on this post refreshed 11 October 2017

1 Comment

Filed under Mentoring & Other Business