Navigation

• Wiki Home Page

• Categories

Administration Methods

• Budget
• Time needed to develop and conduct the assessment
• Practicality
• Political situations

• Document Analysis
• Existing Data
• Focus Groups
• Interviews
• One-Minute Assessments
• Portfolio
• Surveys
• Visual Methods

Document Analysis

• Public Records – the official, ongoing records of an organization’s activities
  • Examples include: student transcripts, mission statements, annual reports, policy manuals, student handbooks, strategic plans, syllabi
• Personal Documents – first-person accounts of an individual’s actions, experiences, and beliefs
  • Examples include: calendars, e-mails, scrapbooks, blogs, Facebook posts, duty logs, incident reports, reflections/journals, newspapers
• Physical Evidence – physical objects found within the study setting (often called artifacts)
  • Examples include: flyers, posters, agendas, handbooks, training materials

• Documents are often readily available on campus. (Stage, 1992)
• Documents already exists, therefore data collected is unobtrusive.
• Data is already “collected” through the creation of the document’s content.
• Since the documents are already created, there are low costs associated with completing the project.
• Documents are stable, once they are created they usually do not change (one exception may be websites).
• Documents are viewed as precise.
• Easy to carry out a document analysis on a quick timeline.
• No coordination is needed between researcher and participant therefore analysis can be done on researcher’s timeline. (Schuh, 2008)

• Access to documents may be restricted or protected.
• Documents may be incomplete or inaccurate.
• Documents are context and language specific, the person doing the analysis should be familiar with the culture of the organization.
• Documents are not interactive so you can’t ask them for more information.
• Documents are often disconnected from their creator so you cannot ask questions about the context or insider knowledge.
• Documents are often written from a specific lens or angle, therefore they lack objectivity.

• Primary Data Source: provides direct information about student life, activities, and institutional processes
• Secondary Data Source: to supplement interviews, observations, and quantitative analysis
• Prompt: to generate questions for interviews, surveys or identify events to be observed

Existing Data

• Tracking or usage numbers
• Survey data (raw or summary)
• National Instrument data (e.g., NSSE, CIRP, NASPA Consortium, EBI)
• Student information system data
• Systems that track student activities and involvement (e.g., CollegiateLink)
• Data from documents such as applications, program evaluations, journals, incident reports, or duty logs
• Judicial system data

• Do not have to take the time to collect data (saves time and money)
• Known response rate since the data is already collected
• Helps to avoid “survey fatigue”
• Encourages collaboration between units in order to share data
• Customize information to meet your specific assessment need (if pulling from a database)

• Gaining access to data if the source of the data is not in your direct control
• Getting data in the form you need it – Administrative systems are built for smooth transactions, getting data from them can be cumbersome.
• Getting your data from a trusted source since reliability of the data is reliant on the source
• Data is non-responsive, therefore you cannot follow-up for more information
• Ensuring student privacy – Analyze data in a manner that does not allow for individual identification.
• Knowing what data is available to use

Focus Groups

• Qualitative information on its own to inform direct practices for a smaller, specific group
• As a pre-study for a larger project, ensuring more accurate questions and phrasing to collect the desired data
• As a post-study activity, gathering more information about the previously analyzed data

• Determine the purpose - This will be when methodology can best be determined as well.
• Determine the population - Who can give you the information you want?
• Draft the questions - This will develop through several methods, including brainstorming, phrasing, sequencing, piloting, and revising.
• Get IRB approval if necessary - This must be determined based on your campus’s IRB policies and the particular methods and purposes of your project.
• Select appropriate numbers of groups, times, and locations - Ideally this will increase your participation.
• Advertise and recruit participants - Point efforts to get the people who can specifically give you the information you want.
• Train moderators and scribes - You will want to know that all participants understand their roles and what they should do before, during, and after the focus groups.
• Conduct the Focus Groups
• Compile the data - Use transcribed sessions, observational data collected, and any other compiled data.
• Analyze the results - Determine the best way to analyze the data, typically some form of theme compilations.
• Write the report - This could include presentations or varying forms of reports that would be most appropriate to each different constituency, which will likely mean more than one type of report. (Krueger, 2009)

• In paper form, one transcript can be cut into pieces for each quote and moved into groupings of themes to piece back together.
• In electronic form, software packages can help you theme the document copy of transcripts. Such document analysis software includes The Ethnograph, Atlas.ti, and NVivo.
• In electronic form, software can also theme by marking portions of audio files or group themes of audio clips. Sound Forge is an audio file analysis tool. (Krueger, 2009)

• Focus groups are meant to start discussions among its members.
• The unstructured, free-flow format is appealing to respondents who feel comfortable talking in front of others.
• There is little chance that members will be confused by a question or topic, as moderators can go back and repeat or clarify a question.
• Visual aids (e.g., charts, posters, video clips) can be used.
• It is thought that focus groups provide believable answers and results that are high in validity.

• Focus groups are not meant for large sample sizes. Rather, they are qualitative and tend to concentrate on a smaller sample and more specifically on human behavior.
• The facilitator and the analyzer need to be certain of their personal biases and attempt to eliminate those as much as possible before conducting the groups or analyzing the data. It could be easier to determine themes or seek themes that fit prejudices or biases about the issue.
• Members of the focus group could either be naturally shy or feel uncomfortable discussing subject matter that can be deemed provocative in front of others.
• Members may not want to “go against the grain” as he or she may suffer from the group dynamic and feel an amount of peer pressure.
  • They may not want to be singled out if they feel a different way, essentially forcing them to provide a dishonest answer

• Little control over group
• Lack of confidence
• Unknowledgeable when it comes to subject matter
• Does not create comfortable and inviting atmosphere

Interviews

One-Minute Assessments

• Minute Papers (that respond to one prompt)
• One-sentence summary of the session
• Muddiest point of the session
• Word Journals (list words related to the session’s topic)
• Direct paraphrasing (summarize or paraphrase the session’s content)
• Application (ask to apply content from session)

• What is the most important/significant thing you learned today?
• What did you learn today?
• What did you learn about __ today?
• What question(s) do you still have? What do you feel requires further clarification?
• What worked best for you about what we did in class today/what worked least well?
• What was the most important point we discussed today/this week?
• Summarize what we just did or learned about.
• What was the clearest point/muddiest point about today’s material?
• Why is this concept relevant to your experience as a …..?
• What is wrong with the following statement . . .?
• Define a key term that has provided a focal point for today’s discussion.
• Make a drawing/graph that illustrates (utilizes) a concept.
• Write a metaphor illustrating a concept that we have focused on today.
• Provide an example test question from the material you have covered today.

• Provide a quick summary of key points from students’ perspective
• Quickly identify areas of weakness and strength
• Are able to track changes over time
• Provide non-verbal feedback from quieter students
• Use students words to describe programs and services
• Allow for a short time commitment: “one-minute”
• Provide immediate feedback about program for improvements
• Are easy to administer using paper and/or clicker systems

• Limited amount of follow-up with students resulting in non-responsive data
• Lack of specific information due to time constraints – Use specific prompts helps to alleviate this risk.
• Difficult to interpret short responses
• Need to collect data over time to see progress/growth in a program
• Plan logistics of how/when to collect and compile well and in advance

Portfolios

• Shows progress/development
• Promotes student self-evaluation, reflection, and critical thinking
• Reflective in nature
• Shows progress over time
• Measures performance based on genuine samples of student work
• Provides flexibility in measuring how students accomplish their learning goals
• Enables teachers/supervisors and students to share the responsibility for setting learning goals and for evaluating progress toward meeting those goals
• Gives students the opportunity to have extensive input into the learning process
• Facilitates cooperative learning activities, including peer evaluation and tutoring, cooperative learning groups, and peer conferencing
• Provides opportunities for students and teachers to discuss learning goals and the progress toward those goals in structured and unstructured conferences
• Enables measurement of multiple dimensions of student progress by including different types of data and materials

• Requires planning ahead and clearly stated outcomes, expectations, etc.
• Requires extra time to plan an assessment system and conduct the assessment
• Development of a systematic and deliberate management system is difficult, but is necessary in order to make portfolios more than a random collection of student work.
• If using for assessment you need trained evaluators and time for the evaluators to work
• Storage and security of portfolios (both on-line or hard-copy)
• Gathering all of the necessary data and work samples can make portfolios bulky and difficult to manage.
• Scoring portfolios involves the extensive use of subjective evaluation procedures such as rating scales and professional judgment, and this limits reliability.

• Provides tangible evidence (the proof is in the paper)
• Gives students visuals to use for interviewing
• Requires storage space on campus with access for students and evaluators
• Limited to one copy of the portfolio
• Limited to 2D items

• Provides easy access by multiple people for multiple purposes
• Able to be linked in resumes and graduate school applications as well as assessment reports
• Easily stores multimedia projects
• Easily altered (can be a strength and a challenge)
• Requires technology systems and storage (may be costly)

• A clear educational AND assessment purpose
• Pre-set determination of outcomes to be demonstrated
• Pre-set criteria for judging quality/success of the portfolio
• Determination of experiences to be included (usually a checklist or rubric)
• Reflection

• What are the goals of the portfolio?
• Who will be creating the portfolio? Who is interested in their results?
• How and when will students choose what to put in the portfolio?
• How will students reflect on the portfolio?
• How will the portfolios be evaluated? Who will review?
• How will the process be kept manageable?
• Who “owns” the portfolio? (especially important for hard copy)
• What are the benefits?
• What are the areas of concerns?

• Proficiency/Document – collection that shows specific levels of achievement
• Best Work/Showcase – collection that shows various “best works”
• Progress/Growth – collection that shows a progression of work (improvement)
• Memorabilia – collection that shows memories of experiences
• Skills – collection that shows specific pre-determined skills

• Table of contents

• Checklist of items to include
• Introductory statement
• Examples/evidence of work (dependent on goal)
• Reflection on each item
• Overall reflection page
• Evaluation summary (usually score card or rubric)
• Evaluation comments (important if formative assessment)

Surveys: Online and Telephone

• Web-based surveys are essentially paperless.
• Delivery of the surveys themselves, along with the results, can be made easy by sending out an e-mail to the desired respondents.
• Survey takers can typically respond at their own convenience.

• Respondents may not be familiar with technology. Furthermore, not everyone has Internet access or access to a computer.
• Problems can be experienced with technology (e.g., faulty Internet access, sluggish computer). It is important to note that with technological advances becoming more ubiquitous, these issues are becoming less significant.

• Intended target may think the e-mail is “spam”
• Size of survey may be too large to administer via e-mail
• Undeliverable (bounce-back) e-mail addresses

• Lack of visual material
• May be considered impersonal
• Respondents may feel annoyed or interrupted by the telephone survey
• Losing ground due to increases in call screening, hang-ups, refusals, and decreases in land-line phones

Visual Methods

• Scientific images – These images are used for cataloguing and include examples such as physical layouts of buildings, architectural change, and studies on time.
• Narrative images – These images tell stories and include examples such as photo essays.
• Phenomenological images – These images capture a subject through the experience (or lens) of the source of the image. Examples include photo journaling where a student captures an image and then writes their views on why the image is important.
• Reflexive images – These images are taken by the person doing the assessment. The key element in type is that the image is being captured and interpreted by the same person. Examples of these may be photos taken by an event planner at each stage in order to assess the event planning process.

• Provide more detail (“a picture says a thousand words”)
• High levels of student investment of time and thought to creating images
• Can use the images captured for other uses such as marketing materials
• Made fun and easy due to availability of digital cameras, flip cams, software, and web technology

• Availability of digital cameras, flip cams, software and web technology make images easy to alter, therefore a threat to your trustworthiness
• Sample size is usually small, therefore you are collecting a smaller number of perspectives
• Additional time is needed to planning and collecting data