Cut to the Core of Me - Online Personals Watch: News on the Online Dating Industry and Business

« Social Neworking via Cell Phones | Main | Riding the Networking Wave »

Comments

James Houran

Dear List,

Despite their potential power and value, all assessments have limitations. Both online dating sites and their customers need to have realistic expectations about what types of information assessments can and cannot deliver. The strengths and limitations of a given assessment are based on its technical and theoretical underpinnings.

Below are some important points to remember in this respect:

1). Assessment feedback is derived from mathematical extrapolations of behavioral data. As such, feedback reports describe statistical predictions of what attitudes and behaviors a given test taker will likely exhibit. Mathematical models are consistently more valid than subjective observations, but even the finest assessments are never 100% percent accurate 100% of the time.

2). The validity of a report is limited by the reliability of the test taker’s responses. Test-takers may answer assessments unreliably for a myriad of reasons: lack of motivation or interest due to less than ideal testing conditions or test taker’s mood, fatigue from answering a long set of questions, an attempt to answer questions in a socially-desirable way or difficulty understanding particular questions for linguistic reasons (e.g., when English is not the test taker’s first language).

3). All test scores are statistical estimates. Thus, each score is accompanied by its margin of error [also called a confidence interval or standard of error (SE)]. However, properly constructed employee assessments provide information on the statistical reliability of a particular test taker’s test scores, as well as measure the degree to which a test taker seems to be answering the assessment truthfully.

4). Finally, the quality of an assessment (and hence its feedback) is associated with its methodological and statistical principles:

Self-referential vs. normative instruments: Some assessments provide feedback based simply on how a test taker perceives him or herself. In other words, these instruments describe individuals only in a self-referential way, i.e., against themselves. Examples of self-referential instruments are the Myers-Briggs Type Indicator (the inspiration for PerfectMatch.com's test) and the DISC assessment (offered by Thomas Technologies). By contrast, normative instruments are inter-individual because they describe test takers against a reference group. This approach is significantly more valid than the self-referential approach.

Classical test theory vs. modern test theory: Most assessments on the market today are constructed and validated using classical test theory, which essentially treats all assessment questions as equally weighted “points.” A great example is the assessment offered by eHarmony.com. Such assessments consequently provide a total score that is the sum of those points. This approach has been outdated since 1960. Today, test and measurements experts rely on modern test theory (Item Response Theory and Rasch scaling), which yields unbiased, scaled scores for test takers. Modern test theory is the same gold standard statistics used in such well-known assessments like the GRE, MCAT and LSAT. This approach can identify and remove response biases related to age, gender, cultural background and employment level of the test taker. Besides greater technical precision and the protection of meeting legal requirements, modern test theory also yields richer information that traditional approaches miss.

For detailed scientific information on the realities behind compatibility testing, see:

Houran, J., Lange, R., Rentfrow, P. J., & Bruckner, K. H. (2004). Do online matchmaking tests work? An assessment of preliminary evidence for a publicized ‘predictive model of marital success.’ North American Journal of Psychology, 6, 507-526.

For a lay-person's guide to the subject, see:
http://www.onlinedatingmagazine.com/features/compatibilitytesting.html

Thanks,

James Houran, Ph.D.
Online Dating Magazine

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Subscribe Here

Research

  • Dig Deeper - Research Categories

We're Social

  • Facebook  Twitter   Youtube Linkedin