SV United StatesSV USA SV CanadaSV CAN

Assessment Research and Terminology

Modified: 2010/08/19 15:57 by admin - Uncategorized

Edit

Quantitative Research

Quantitative research is a mode of inquiry that attempts to systematically measure and predict phenomena through the use of standardized tools. It is inherently a world of numbers, percentages, and therefore tangible analysis.

Quantitative researchers tend to approach the world as a computable place. Data is possible; they just need to go find it. They tend to follow the deductive method, which begins with general principles and moves towards specific conclusions. This approach contends that information (or knowledge) is everywhere, and we must purposefully corner off bits and pieces of it to ever know reality in its entirety. (1)

Since quantitative research is founded in a positivist school of thought, it is heavily reliant on standardized tools of inquiry. Positivism argues that research must be conducted and found in the “real” world through the human senses (and instruments), thus it is no surprise that the first form of quantitative research was modeled for the physical sciences. Gustav Fechner was the first to model a quantitative approach to psychology in his work in psychophysics. (1)

While we generally do not think of statistical data as “physical” evidence, the definition of data has evolved over time. Counting is thought to be the first of all quantitative efforts, and we have come a long way since the primary forms of quantitative assessment. Rather than measuring physical entities, assessment in the social sciences typically attempts to record phenomena.

  • Designed to gather information and test a particular hypothesis or theory
  • Uses simple to complex statistics
  • Gathers numbers or numeric values to represent categories
  • Reported in tables or numbers
  • Often seeks to generalize to the greater population with larger samples
  • Often uses less time and financial resources compared to qualitative methods, due to the availability of many technologies
  • Matches well with outcomes about knowledge and comprehension (define, classify, recall, recognize)

Examples of tools used to collect quantitative data:
  • Survey
  • Usage numbers
  • Rubrics (if assigning #s)
  • Tracking numbers

Validity Reliability[1]:
  • Validity
    • Asks is your instrument measuring what you think it is measuring
    • Looking for accuracy of your findings
  • Reliability
    • Asks are the answers consistent (meaning if the instrument is used repeatedly you would get about the same answers every time)
    • To what degree are the findings consistent.
    • Looking for consistency to the findings.

Edit

Assumptions of Quantitative Research

Some of the key philosophical assumptions of quantitative researchers are as follows.

  • Reality is single, fragmentable, and tangible.
  • Social facts have an objective reality.
  • Causality is traceable.
  • Variables within an experiment can be isolated, identified, and controlled. (1)

Quantitative research aims to generalize and find a consensus or norm. Thus, variables, especially those found in the relationships between proposed causes, are controlled through research design. Quantitative researchers typically aim to minimize the statistical hindrances variances cause through the implementation of constants or controls within their experimentation. (3)

Edit

The Quantitative Method

The quantitative method closely follows the process of the scientific method. Simply put, quantitative researchers typically:

  • Form a hypothesis or theory
  • Conduct their research using valid and formal instruments
  • Generalize about their findings

Oftentimes, generalizations within an experiment are used as jumping-off points for further quantitative investigation.

Edit

Quantitative Results

Since quantitative researchers are interested in predicting and measuring phenomena, they are most interested in finding a consensus or norm. Likewise, they often have a large pool of results to help verify this norm. Within this large pool of results, quantitative researchers can determine whether or not their hypothesis is correct. Since quantitative research is deductive in method, a well controlled experiment or survey will ideally glean results that prove or disprove the hypothesis, rather than offer completely unexpected data. This is quite different from qualitative results, which can evolve significantly before the experiment is complete. Quantitative results are most often presented in numerical indices and the write-up is typically in abstract language. Because quantitative results are by nature mathematical, the results are often presented in graph form. (2)

One of the most fundamental laws in statistical quantitative analysis today is the notion that correlation does not necessarily indicate causation. It is quite easy to graph a variable that appears to be having direct effect on another variable, but it is always possible that their relationship is invalid (a spurious relationship), and that both are affected by an unknown variable (moderating variable). (9)

Edit

Daily applications

Most researchers do not have the time on their hands to grapple with the philosophical implications of their research methods. They have deadlines or presentations to prepare and at times must limit the extent of their data analysis. In a perfect world, researchers would begin with a hypothesis or theory, use a reliable research instrument, and take time to thoroughly analyze their results.

Edit

Qualitative Research

Qualitative research is a mode of inquiry, which aims to understand and describe phenomena. In contrast to quantitative research, qualitative research and assessment typically involve smaller, more focused samples, where the researcher analyzes subjects inside their contextual environment. This research methodology is primarily used within the social sciences, and looks to investigate and understand the forces that guide human behavior. Hence, it generally takes the form of case studies, observational analysis, and one-on-one interviews. Qualitative research is founded in the naturalist pool of thought, which attempts to examine the environmental (and hereditary) influences of human behavior. They tend to use the inductive scientific method which begins with specific observations and moves toward general conclusions. (1)

  • Researcher(s) gathers information from the world around them
  • Researcher(s) make multiple constructions and interpretations of the world they are studying
  • Gathers text/narrative from respondents
  • Focuses on more in-depth information, digs below surface data
  • Has the ability to capture “elusive” evidence of student learning and development
  • Consists of a small more specific sample or set of respondents
  • Matches well with outcomes about application, analysis, synthesis, evaluate

Examples of tools used to college qualitative data:
  • Interview
  • Focus Group
  • Portfolios
  • Rubrics (if descriptive)
  • Photo Journaling

Standards of Rigor (or validity/reliability of data)
  • Standards of rigor are different from, but no less mandatory than, those used to judge quantitative research
  • Standards of rigor focus on Trustworthiness of the data and include whether the data has[2]:
    • Credibility
      • Are the researcher’s interpretations credible to the participants?
    • Transferability
      • To what extent are the findings applicable to other settings, situations, etc.?
    • Dependability
      • To what extent were the methods decisions made consistently and appropriately throughout the research process?
    • Confirmability
      • Do the results of the study make sense?
      • Can they be confirmed by others?

Edit

Assumptions of qualitative research

Some of the key philosophical assumptions of qualitative research are as follows.

  • Reality is a social and subjective construct.
  • Variables are too interwoven to measure, especially without a contextual framework.

Since qualitative research aims to investigate and understand decision making, intentionality, and the variables within a contextual environment, researchers believe social theory can only be attained when the human instrument is immersed within the context of its subjects. This is why the researcher will most often directly interview research subjects, hoping to gain insight into their personal context and the forces that guide their behavioral patterns. Qualitative researchers, oftentimes those in the mental health field, will approach their subjects with empathy and partiality to elicit genuine responses. (1)

Edit

The Qualitative Methodology

Typically, qualitative researchers rely on four methods to collect data.

  • Participation in the setting
  • Direct observation
  • In-depth interviews
  • Analysis of documents and materials (6)

Other more specific qualitative approaches include focus groups, mixed method, case studies, narratology, phenomenology, grounded theory practice, ethnography, and storytelling. (6)

Edit

Qualitative Results

Qualitative researchers usually do not acquire extensive research and documentation that they are able to graph numerically. As such, their write-ups are vastly more descriptive than quantitative ones. Within this write-up, qualitative researchers are responsible for conveying and translating the social theory they have witnessed to their peers. Since this research methodology is largely contingent on the subjectivity and interaction of the researcher with the research subjects, the write-up is where they argue their theory and offer proof.

Edit

Grounded theory

Grounded theory was developed in the late 1960s by Barney Glaser and Anselm Straus. It emerged at a time when qualitative research was deemed too unscientific, especially in the wake of quantitative popularity. Grounded theory is a slightly more scientific method of identifying causal relationships within qualitative data than the more traditional approaches such as storytelling and ethnography. Nonetheless, it was somewhat controversial when it first emerged onto the scientific scene, as it completely reverses the deductive process of the scientific method. Grounded theory begins by analyzing a large corpus of data in the hope of finding and labeling specific contextual variables. Oftentimes, since this approach is rooted in the qualitative method, data takes the form of fieldwork such as observational notes or diary entries. From here, phenomena within the data are codified, and the codes are then analyzed for relationships.

The way in which codes are sorted and related also depends on the aim of the research. The way in which a corpus of data is analyzed is also heavily contingent on the attitude and ability of the researcher, and known as “theoretical sensitivity.” The relationships among codes are used to generate theory. Ideally, the generated theory will fit one set of data perfectly, primarily because it is created—ground up—from that very data. (7)

Edit

Daily applications

Theoretically, qualitative and quantitative research are at polar ends of the assessment spectrum. However, these methodologies are frequently used to supplement one another (mixed method). Surveys, for example, which are often considered a quantitative tool, frequently include open-ended questions. Since open-ended answers are not structured to elicit specific results, the process of interpreting and analyzing open text is, by nature, qualitative.

In higher-education assessment, particularly within student affairs, surveys are an increasingly popular tool. However, there are several other assessment methods that are more qualitative in nature. Focus groups, interviews, and written reflections are all examples of qualitative methods.

Edit

Direct and Indirect measures

Direct Methods:
  • Any process employed to gather data which requires students to display their knowledge, behavior, or thought processes for the assessor
  • A strength of a direct measure is that it is capturing a sample of what the student can do[3]
  • A weakness of a direct measure is that not all things (such as values, perceptions, feelings and attitudes) are easily captured
  • Data is often gathered through tests/quizzes, rubrics, portfolios, oral presentations, case studies or other authentic measures
  • Example in student affairs: Where on campus would you go or who would you consult with if you had questions about which courses to register for the fall?

Indirect Methods:
  • Any process employed to gather data which asks students to reflect upon their knowledge, behaviors, or thought processes. It is a report of perceived learning.
  • A strength of indirect measures is that they can provide additional information and feedback about what is valued[4]
  • A weakness of indirect measures is that is weaker measure of student learning
  • Data is often gathered through surveys using agreement, extent and satisfaction scales
  • Example in student affairs: Please rate your level of agreement with the following: I know of resources on campus to consult if I have questions about which courses to register for in the fall. (Scale: Strongly agree, Moderately agree, Moderately disagree, Strongly disagree)

Edit

Population and Sample

Population:
  • Used in statistics to represent all possible measurements or outcomes that are of interest to us in a particular study.
  • Examples: Cedar Crest students; trees in North America; automobiles with four wheels; people who consume olive oil[5]

Sample:
  • A portion of the population that is representative of the population from which it was selected.
  • Examples assuming the populations stated above: 47 Cedar Crest students chosen randomly; 8463 trees randomly selected in North America; 20 sample autos from each make (e.g., GM, Ford, Toyota, Honda, etc.); 1% of the oil consuming population per country[6]

Edit

Sampling Methods

For most surveys, a sample or target population is necessary, as the population is too large to survey in its entirety. However, if the population is small enough, it is possible to survey each member. This type of survey is known as a census study. 1

With larger-scale surveys, samples are almost always a necessity. The two main types of sampling methods are probability and non-probability sampling.

There are various types of samples that one can use depending on the type of assessment they are trying to conduct.

Edit

Probability Sampling

The probability sampling method is based on the likelihood that each member of a population has an equal chance of being selected to be in the sample. Most researchers agree that this form of sampling is the closest to representing the actual population, as human bias is eliminated with the use of computational randomization. One of the key advantages of probability sampling is that it is the easiest to measure for error. Probability sampling methods include:

  • Random sampling is the truest form of probability sampling. This type of sampling guarantees that each member of a population has an equal chance of being included in the sample. This type of sampling is ideal for more controlled studies where human bias in the selection of the sample is intolerable.

  • Stratified sampling is a more sophisticated version of random sampling that starts by determining the various stratum (i.e., subset, division) within a population and then drawing randomly from each stratum so as to not exclude or misrepresent any one stratum. Examples of popular stratum include the various ethnicities or age groups within a population.

  • Systematic sampling, also called nth name selection technique, is often used instead of random sampling due to its simpler process. After the sample is collected, every nth member within the population is recorded within the sample. For example, if a mentor wanted a systematic randomized sample from a list of mentee evaluations, she could select every 7th evaluation and analyze this extracted sample as a more random representation.2

The main drawback to systematic sampling is that the set pattern (e.g., every 7th time) may coincide with another underlying pattern within the sample. For example, the mentor may be selecting every 7th evaluation at random, but if she had separate mentoring sessions for every seventh mentee, she would be selecting the first evaluation from each session each time she picks.3

  • Multi-stage sampling essentially combines the techniques of several sampling methods.4

Edit

Non-Probability Sampling

Non-probability sampling is often used in more experimental or trial research and does not guarantee a random and close representation of the actual population. Rather than using computational or systematic randomizations, non-probability sampling employs human judgment and often relies on sheer convenience.5

Some of the various types of non-probability sampling include:

  • Convenience sampling is used when researchers need a cost-effective estimate of the data one would find from doing a more randomized sampling. Typically, these studies are used as jumping off points for larger more comprehensive studies with more representative samples. Also, as the name suggests, the samples chosen are pulled primarily because they are convenient.6

  • Judgment sampling relies heavily on human judgment to select a sample that is an appropriate representation of the population. Similar to convenience sampling, judgment sampling relies on human judgment often because more effective resources are too expensive or unavailable.7

  • Quota sampling is comparable to stratified sampling in that the researcher first identifies the various stratums in a population. Contrary to stratified sampling, quota sampling then employs either convenience or judgment sampling to select the members within each stratum.8

  • Snowball sampling employs current sample members to recruit additional members within their specific population through word-of-mouth. Once the recruitment process begins and sample members sign-on to the study, the effect is similar to that of a snowball effect.9

  • Snowball sampling is often used within unique or unapproachable populations, which are difficult to thoroughly assess otherwise.10

Edit

Response rate

Response rate most commonly refers to the rate at which a survey sample responds to the administered survey. Mathematically, it is the number of respondents to a survey, divided by the number within the sample. The response rate is often expressed as a percentage.

For example, if there were 500 people in a survey sample, and only 320 responded, the response rate would be 64%. (1)

When you divide 320, by 500, the number generated will be a decimal. To create the percent, simply move the decimal over two places to the right, such that .64 becomes 64%.

Edit

Sample size

The larger the sample size, the more likely it is that one has selected a representative piece of the population. Consequently, the results from the survey are less likely to be due to chance. With a larger sample, researchers can also more confidently generalize about their findings to the public and argue that they have identified a social norm. Researchers commonly create very large sample sizes to reduce the margin of error within their results.

Factors to consider in your sample size:
For information on sampling, see Types of Sampling.

For more information on statistics and measuring results, see Standard Deviation, Standard Error of the Mean, Confidence Intervals.

Edit

Correlation and Causation

  • "Correlation does not imply causation" is a phrase used in science and statistics to emphasize that correlation between two variables does not automatically imply that one causes the other[7]
  • Both are statistical techniques used to test the relationship between variables or factors
  • Correlation measures whether a relationship exists between two variables and if there is a relationship, it also measures the strength and direction of that relationship.
  • Causation tests the hypothesis that one variable has an effect on another variable
  • Example: Look at the two variables of alcohol use and risky sexual behavior. These two variables are often found to have a strong positive correlation (meaning as alcohol use increases so does the risk of sexual behavior. Therefore these two variables are correlated. However, saying that alcohol use causes risky sexual behavior is not valid as there are several situations and reasons by risky sexual behavior may occur.

Edit

Types of Survey Questions

There are essentially two basic types of survey questions: open-ended and closed-ended.

Edit

Open-ended questions

Open-ended questions tend to be fairly self-explanatory, as they ask the survey respondent to answer a question in their own words. They are also, in general, easier to formulate, as you do not need to come up with an exhaustive list of answer choices.

Open-ended questions tend to lean more towards the qualitative research spectrum. As such, they are difficult to assess through statistical analysis software and require a human eye to categorize and quantify results. (1)

Edit

Closed-ended questions

Closed-ended questions can significantly range in style and content. There are approximately six types of basic closed-ended questions.
  • Dichotomous
  • Categorical
  • Ordinal
  • Numerical
  • Multiple-choice
  • Rating scale questions (Likert-scale/style) (1) See Scaling Choices

Edit

Dichotomous

Dichotomous questions are simply questions with Yes/No answer choices. They are the simplest of all the closed-ended questions, and as such prove extremely easy to answer. (2)

Dichotomous questions are often adapted into Likert-scale ranges if the statements to be rated are likely to have a spectrum of possible answers. In other words, dichotomous questions, when applicable, are a great and direct survey tool, but a simple Yes/No answer set can rule out the “shades of gray” within more opinion-based or complex questions.

Edit

Categorical questions

Categorical questions are questions that asks respondents to identify themselves within a category. These are most frequently used in the demographic sections of surveys. A common example is an academic ranking question, where the answers could be:

  • Freshman
  • Sophomore
  • Junior
  • Senior

Respondents do not have to thoroughly consider the answers in a categorical answer set; they simply fit into one category. Of course, it is always prudent to include an Other (please specify) option on a survey question, but this may not be necessary if the survey administrator knows whom they are surveying.

Edit

Ordinal questions

Ordinal questions can also be described as ranking or rank order questions, and these typically ask students to rank a list of items from highest to lowest. The type of order depends on the survey designer’s aim. A popular type of ordinal question is degree of importance, for example:

Of the following list, please rank the top five traits you believe are most important for a leader to possess, with 1 = Most important

  • Camaraderie
  • Charisma
  • Courage
  • Honor
  • Humor
  • Intelligence
  • Prudence
  • Skill
  • Tenacity
  • Wit

The list of possible ranking items does not have to be limited to the number of items you want ranked. You could have respondents rank all 10 of the above options, but the number of items to be ranked are often recommended to be kept shorter, so that respondents are not overwhelmed by a multitude of options.

Edit

Numerical questions

Numeric questions are simply questions that require a real numeric answer. It is important to word numeric questions as specifically as possible so that the desired number and format are collected. Popular examples include age and date questions. Numeric questions can be structured as open-ended, but may be formatted with specific requirements (e.g., required to enter a 2 digit number) if the survey is administered via the Web or other technological device.

Edit

Multiple-choice questions

Multiple-choice questions essentially ask respondents to choose one or more answers from an established answer set. It is within these questions that options like Other (please specify) and None of the above are important, so as to create an exhaustive set of answers.

Unlike categorical questions, multiple-choice questions may involve some degree of opinion, especially if respondents are limited to only one response. For example:

Which of the following issues is the most important to you as a college student today?

  • The war in Iraq
  • Global climate change
  • The economy
  • Immigration
  • Other (please specify)

Multiple-choice questions can also be completely non-opinion based and simply ask a question, for which the majority of possible answers are pre-conceived. For example: How did you hear about this event? (Check all that apply)
  • Newspaper
  • TV
  • Professor
  • Word of mouth
  • Other (please specify)

Edit

Scaling choices

Scales can range significantly in style and intent. Essentially, scaling questions ask respondents to rate various items or statements with respect to their level of agreement or attitude. Among the most popular rating questions are the Likert-scale/style questions.

Edit

Likert-scale questions

Likert-scale questions specifically deal with agreement scales. These questions usually list a series of statements and ask the respondent to rate their level of agreement with each. (1)

Typically, agreement scales range from strongly agree to strongly disagree, however, the order in which scales are presented—whether ascending or descending—is typically left to the discretion of the survey designer. Scale labels themselves, as well as the number of points on a scale can also differ significantly. The most common scale is most likely the five point scale, which includes a neutral center point and then two degrees of positive and two degrees of negative on either side. With that said, three, four, six, and seven point scales are also regularly used.

Likert-style questions are based on the Likert-scale, but include all the other types of scale labels, such as excellence, satisfaction, and importance. While the Likert-scale questions typically use bipolar scale labels, Likert-style questions often use unipolar scales.

Edit

Bipolar scales

Bipolar scales, as the name suggests, extend across a spectrum that contains both a positive and a negative pole. For example, strongly agree, moderately agree, neither agree nor disagree, moderately disagree, and strongly disagree

These scales often have a neutral middle point, but it is not always applicable or necessary. Some researchers in the assessment field believe that this neutral point can keep respondents from answering as truthfully as possible, by allowing them to opt out.

For the most part however, those in the assessment field agree that a scale containing a “not applicable” option should also contain a neutral point, so that those who have a neutral opinion on the topic are not incorrectly choosing “not applicable.”

Edit

Unipolar scales

Unipolar scales extend across a single pole and include more specific degrees of descent, than the bipolar scales. For example, a unipolar usefulness scale could include the following labels: Extremely useful, Very useful, Moderately useful, Slightly useful, and Not at all useful. The scale descends along smaller degrees of usefulness than would a bipolar scale.

Unipolar scales are helpful in determining the specific extent to which something is useful, important, helpful, etc., whereas in some cases it might be pointless to assess the degrees to which something is not useful, important, etc.

Edit

Dichotomous question

The dichotomous questions, or “yes/no” questions, are often grouped together to convenience the survey taker. When in group form a respondent does not need to scroll through a long series of single “yes/no” questions, but can check them off in sequence. Sometimes, dichotomous scales are used to replace “check all that apply” questions, as it is thought to be easier for respondents to identify their relationship to an item if they can think in terms of yes or no. (1)

Edit

Measures of Central Tendency

Measures of central tendency are statistical methods used in mathematics, which attempt to find the most common or center most point in a set of data, and thus help to summarize or typify the data within the set. The central point is a value measured in a way that provides a fairly close estimation of the norm. Locating and understanding this center point is essential in quantitative assessment, which typically attempts to locate and predict normality within data. (1)

The three most well known measures of central tendency are the mean, median, and mode.

Edit

Mean

The mean, also known as the average, can be easily described as the sum of all the values in a set, divided by the number of values within the set. For example: 5 + 7 = 12, 12/2 = 6, so the mean of 5 and 7, is 6.

Edit

Median

The median is found by placing the values within a set into ascending order. If the number of values is odd, the median is simply the middle number, dividing the set into two even halves with the median in the middle. If the number of values within the set is even, the median is the mean of the two center most points. For example: {10, 12, 15, 16, 18, 19} This is a set of data with an even number of points, so to find the median, we must find the average of 15 and 16, which is 15.5.

Edit

Mode

The mode is measured by finding the most frequently repeated number within a set. For example: {1, 3, 3, 5, 8, 9, 10} The mode within this set is 3.

Edit

Standard Deviation, Normal Distribution, Standard Error of the Mean, Confidence Intervals

Edit

Standard Deviation

Standard deviation measures the degree to which data deviates from the mean. Since points of a data with a set are all components of the mean, standard deviation also helps determine the degree to which data points differ from one another. In other words, standard deviation quantifies scatter within a set. Standard deviation is important to consider if the data points within a set are extremely scattered. If this is the case, the median may be a better measure of normality.

To determine the standard deviation within a set, you first subtract the mean of the set from each individual point in the set. There is now a new set of numbers. Then each new number (data point – mean) is multiplied by itself (squared). Finally, all these newly found squares are added together and divided by the total number of points in the set minus 1. (3)

You can also use Excel to compute the standard deviation.

Edit

Normal Distribution

In most data sets, data points tend to fall near the mean. This phenomenon, in statistics, is known as normal distribution. The theory of normal distribution claims that in a set of finite variables the majority of the data points will fall near the mean.

Normal distribution is also discussed with reference to the Central Limit Theory, which essentially states that a large enough sample size of data will typically have a normal distribution.

A common example of normal distribution is found in national health averages. Most people tend to have a height that is close to the national average and few tend to stray too far from the mean. Visually, normal distribution takes on the bell-shaped curve, which heightens in the middle and slopes down on either side. The curve is steeper if the data points are less scattered and fall closer to the mean, and less so if the points vary from one another and deviate greater from the mean. (4) (Insert Picture)

Edit

Standard Error of the Mean

Standard error of the mean (SEM) accounts for the difference between the values of the estimated mean (within the sample) and the actual mean (unknown mean of the population). The estimated mean is inherently error-prone as it cannot be perfectly representative of the entire population. However, the larger the sample size, the more likely it is that the sample is representative of the population as a whole. Also, with larger samples, the Central Limit theory, which visually manifests as the bell curve, is more likely to take effect.

To find the SEM, the standard deviation is divided by the square root of the sample size. As the sample size increases, the SEM decreases. (5) (Insert visual equation)

Edit

Confidence Intervals

Essentially, confidence intervals measure the margin of error in the calculation of the estimated (sample) mean. Though any percentage could be used to calculate confidence intervals, 95% is universally accepted as a mathematically appropriate choice. (6)

As such, a confidence interval of 95% suggests that the data collected from the sample has a 95% chance of containing the mean of the overall population.

To calculate the necessary size of a sample based on accepted error and population size, try the following link: http://www.touchpoll.com/calculator.htm

References

ScrewTurn Wiki version 2.0.33. Some of the icons created by FamFamFam.

© 2010 StudentVoice