ASSESSMENT AND EVALUATION EXAMPLES
1. Program-wide Education Assessment and Evaluation
1.1. Center: Biomimetic MicroElectronic Systems (BMES) ERC, a graduated ERC (2003–2013)
Lead Institution: University of Southern California
Center Director: Dr. Mark Humayun, Dept. of Ophthalmology and Biomedical Engineering
Name of Program: BMES ERC Assessment Examples–Education and Outreach (E&O)
Type of Program: Assessment examples (across all E&O programs)
Program Synopsis: For the past 10 years, the BMES ERC cumulative educational assessment has focused on the following important areas: a) student knowledge gained resulting from BMES ERC-initiated courses; b) preparedness metrics aligned with recommendations of the National Academy of Engineering (Educating the Engineer of 2020, 2005) and the Accreditation Board of Engineering and Technology (ABET Manual, 2011); c) measures of students’ creativity, propensity for innovation, and entrepreneurship; d) measures of preparedness of engineers for a global economy/workforce; and e) impact assessments that focus on the careers of Center alumni. Our education assessment results are presented in these areas for this report. These efforts have addressed both the Generation Two (Gen-2) and Generation Three (Gen-3) ERC requirements outlined in the NSF guidelines. Assessment results are presented for undergraduate, graduate, and alumni activities. Measurement and instrumentation is described for each program focus. The figure below illustrates the cumulative interconnectedness of our ERC education programs and associated assessment metrics.
Contact person/website: Gigi Ragusa (ragusa@usc.edu)
Dates of Operation/Timeframe: The assessment program began in year one of the ERC. It is ongoing and fully developed.
Background: Comprehensive assessment measures and procedures with foci on BMES ERC research areas enable us to assess the effectiveness of and continuously improve our undergraduate, graduate, and K-12 education and outreach efforts. It should be noted that, while Dr. Ragusa was funded under BMES core funds as the staff Assessment specialist, BMES also had a number of other grants to develop these assessments. These included the US Department of Education Institute of Education Sciences (IES) and the Department of Energy’s Energy Frontier Research Centers (EFRC), as well as NSF’s REE, RET, REESE, TuES, and the older CCLI and IEECI programs. While it is important to do assessment and evaluation, it is not expected that ERCs will always develop new tools out of ERC funds, but rather will use relevant, existing tools and other funding where available.
Methodology: These efforts are described in two major categories below: university-level assessments and outreach/precollege assessments. All assessments are psychometrically sound (Wilson, 2012) per Item response theory, valid, and statistically reliable. They are aligned with National Academy of Engineering, National Research Council (NRC), and American Engineering Association (AEA) best practices. All partner institutions in BMES ERC have Institutional Review Board (IRB) clearance for these assessments.
UNIVERSITY-LEVEL ASSESSMENTS
Biomedical Engineering (BME) Concept Inventories: Concept inventories were designed and administered to measure the knowledge gain associated with BMES ERC biomedical engineering courses. A concept inventory is a conceptual measure of students’ course-specific knowledge (Hestenes et al 3]. Concept inventories are administered at the beginning and at the end of a course to measure gains in conceptual understanding as a result of completing the course. These inventories are best used for undergraduate courses that have primary foci on developing conceptual understanding. The concept inventories were aligned with the primary research areas of the BMES ERC (Hestenes, et al, 1992), and were the first developed in biomedical engineering.
ERC Student Research Experiences: We also measured the effects of undergraduate and graduate students’ research experiences on their skills and perceptions of the Center’s research program by administering an interdisciplinary research questionnaire. This was aligned with NAE’s Engineers of 2020 recommendations. These results were compared to a rubric rating of the ERC student research teams’ annual presentations.
The Engineering Global Preparedness Index: In concert with NAE recommendations and guidelines for NSF Gen-3 ERCs, we measured our BMES ERC students’ preparedness to work in global workforces. An 18-item questionnaire with a six-point Likert-type scale was developed. This assessment has also been used at thirteen universities nationally as a measure of achievement of the Accreditation Board of Engineering and Technology (ABET) outcomes F (ethics), G (communication), and H (global preparedness) using workforce preparedness as an assessment frame.
Four subscales are used as constructs in the engineering global preparedness index (EGPI). These scales are aligned with important skills needed by both engineers and other professionals who work in global marketplaces (and aligns with Gen-3 ERC requirements). A description of each EPGI construct/subscale follows.
Engineering Ethics: Depth of concern for people in all parts of the world; moral responsibility to improve conditions and take action in diverse engineering settings (ABET OUTCOME F).
Global Engineering Efficacy: Belief that one can make a difference; support for personal involvement in local, national, international engineering issues and activities towards achieving greater good using engineering technologies (ABET OUTCOMES G AND H).
Engineering Global-centrism: Valuing what is good for the global community in engineering related efforts, not just one’s own country or group; making judgments based on global needs for engineering and associated technologies, not ethnocentric standards (ABET OUTCOME H).
Engineering Community Connectedness: Awareness of humanity and appreciation of interrelatedness of all peoples and nations and the role that engineering can play in improving humanity and meeting human needs. (ABET OUTCOME G)
Engineering Creativity and Propensity for Innovation: Instruments were developed to measure students’ creativity and propensity for innovation. This instrument was designed and tested because of ongoing conversations with engineering educators nationally and the desire to assess the role that comprehensive educational and engineering experiences have in important industrial and academic skill sets: creativity and innovation. This instrument is currently being used at two other ERCs and three additional universities with other programs. This instrument is aligned with several theoretical perspectives on creativity research (Torrance, 1974; Abedi, 2007; and Khatena, 1999). Constructs included in the revised Engineering Creativity and Propensity for Innovation Index (ECPII) are engineering fluency, flexibility, disciplined imagination, originality, and design thinking. These constructs are closely aligned to the combined research on creativity and innovation and have domain specificity to engineering.
ECPII’s five theoretically grounded constructs (measured in subscales) are described below. Problem solving is the focus of this metric.
Engineering Fluency: Students’ breadth of understanding of diverse aspects of the engineering disciplines.
Engineering Disciplined Imagination: Students’ ability to imagine diverse/unique possibilities within engineering disciplines.
Engineering Flexibility: Students’ ability to imagine diverse problem-solving approaches within the engineering discipline coupled with ability to use a diverse engineering problem-solving skill set in the face of distractors.
Engineering Originality: Students’ ability to develop and design problem-solving approaches that are unique.
Engineering Design Thinking: Students’ ability to design with both breadth and depth using an advanced problem-solving approach.
The instrument has 37 items on the ECPII with three to five items per subscale as described above. This item distribution and scale total is supported by item response theory for designing difficult to observe (i.e., soft skill) constructs. A minimum of two items per subscale in the index are: a) reverse-scored items in support of best practices in survey development; and b) true measurement of students’ ability (rather than student perception) beyond what is self-reported. A six point Likert-type scale was employed for the majority of the ECPII items. A final set of items situated at the end of the index are open-ended and include the requirement that respondents read a context and discipline embedded-scenario and solve an engineering problem via a listing or illustration of steps to problem-solving. This subset of items is rated using a six-point checklist aligned with the subscales in the ECPII.
Research Experience for Undergraduates (REU): As assessments for the BMES REU program, both a questionnaire (also used with our graduate and undergraduate students as described above) and an annual focus group with the participants while they were participating in the REU were used.
Alumni Assessment: Assessing and tracking BMES ERC student alumni since the ERC’s conception has been conducted. A survey is done of graduates to assess their career trajectories and includes alumni who took ERC courses, engaged in an ERC lab experience, or both. They were also followed via social working sites.
Education Assessment Summary: Our cumulative assessments across ERC education programs continue to reveal impactful results. Our ERC students are prepared for industry employment and graduate school and have benefited from the diverse, interdisciplinary education and research activities provided by the ERC. It is important to note that the development of assessment instruments is an iterative process and modifications are often required before the most useful length and format is developed.
OUTREACH/PRECOLLEGE ASSESSMENTS
Science for Life: Content specific concept inventories for our Science for Life program in grades 3-5 were developed. A science and engineering interest survey for students to measure changes in students’ interest in science and engineering across the three grade levels is used. Students’ California Standards Test (CST, an achievement test) scores are tracked across years in school to monitor changes for those participating in the program.
Engineering for Health: In our Engineering for Health program—a high school “Young Scholars” type of program—we also track students, CST scores across years in school to monitor changes for those participating in the program. We also have a high school science motivation scale that we use to monitor changes while participating in the program.
Research Experience for Teachers (RET): A science literacy qualitative inventory to measure participants’ science teaching and performance and students’ (of RET teachers) science motivation and engagement and science literacy was developed and tested for reliability and validity.
Five assessment metrics were used to judge the success of the teacher and associated student intervention programs (RET and California Postsecondary Education Commission Improving Teacher Quality [CPEC ITQ]). These include: a teacher instructional performance measure: science teaching efficacy measure: student science concept inventory: student science literacy measure: and a student science motivation, engagement, and interest measure.
Teacher Metrics
Teacher Instructional Performance Metric: We used a rubric-scored observational assessment of science teacher instructional performance aligned to California’s teacher performance assessment entitled Performance Assessment of California’s Teachers (PACT).
Science Teaching Efficacy Beliefs Instrument Revised (STEBI-R): This instrument is a teacher metric and is a measure that assesses the teacher’s efficacy in teaching science to middle and high students. It includes personal science teaching efficacy and science teaching outcome expectation, and is administered as a pre- and post-test to all teacher participants, then compared to non-participant science teachers that match the participant teachers demographically (based on national averages).
Student Metrics
Science Qualitative Reading Inventory: This metric measure students’ science literacy by grade level. It includes a measure of science vocabulary, reading comprehension, and science writing and is matched in terms of grade level science content and vocabulary.
Grade and Content Specific Concept Inventories: These concept inventories measures of grade level concepts critical to scientific understanding.
Motivation for Science Questionnaire: This questionnaire measures students’ interest, motivation, and engagement in science.
Impact/benefits: This comprehensive suite of assessment provides both formative and summative impact-focused assessments of all education and outreach programs at BMES ERC.
Sustainability: This assessment suite for our programs has been institutionalized at USC and with our partner institutions (university, community college, and K-12).
Tips: Start designing assessments based on research efforts early, obtain IRB clearance at all sites in our ERC partnerships, and work at sustainability/institutionalization early on (at least by year three).
1.2. Center: Center for Integrated Access Networks (CIAN) ERC
Lead Institution: University of Arizona
Center Director: Dr. Nasser Peyghambarian, Department of Materials Science and Engineering
Name of Program: CIAN Assessment Examples–Education and Outreach (E&O)
Type of Program: Assessment examples (across all E&O programs)
Program Synopsis: CIAN is a Gen-3 ERC and, as such, its university education programs assessment must address the requirements delineated by NSF. Essential to CIAN’s systematic assessment plan was to develop early-on clear objectives of CIAN’s education program. CIAN’s assessment efforts measure the extent to which these objectives are achieved by measuring outcomes of CIAN’s education programs, such as increases in knowledge and awareness and changes in attitudes or beliefs. CIAN’s education objective is to engender in its students the National Academy of Engineering’s Engineer of 2020 attributes. This objective drives the activities CIAN develops and nurtures. The outcomes of producing engineers with these skill sets are assessed using an array of tools, such as self-report surveys, interviews and focus groups, graduate student portfolios, and alumni tracking efforts. The figure below shows a quadrilateral paradigm that visually demonstrates the rationale behind CIAN’s program development. It shows how CIAN’s education activities are mapped to the CIAN objective it is designed to engender.
Quadrilateral Paradigm Tied To CIAN ERC Education Objectives
Contact person/website: Allison Huff MacPherson (allison@optics.arizona.edu)
Dates of Operation/Timeframe: The CIAN ERC was founded in 2008.
Background: CIAN’s education programs strive to engender the skill sets consistent with the Engineer of 2020 attributes in its students, as outlined in the figure below. Programs and activities are developed and assessed with this in mind.
Note: For additional relevant information, see Appendix 4.5, part A: Programs to Produce Desired Skill Sets, in the Graduate Education section of the chapter.
Methodology: The types of assessments CIAN uses are described below. Most assessment tools are self-report/survey items. Institutional Review Board (IRB) clearance at each partner university should be obtained.
Formative (or process) evaluation provides ongoing measurement while developing CIAN education activities for the purpose of monitoring and improving the achievement of CIAN’s educational objectives of developing an education program that graduates students who are more effective in industrial and academic practices. Formative tools are used to assess CIAN’s education program and assure that its activities are aligned with the desired skill sets CIAN’s education program is engendering in its students. Results from the process evaluation are used mainly to improve the quality of the activities and reinforce our educational strategy.
Summative (or outcome) evaluation provides a comprehensive evaluation at the conclusion of an activity that measures the achievement of CIAN’s education outcomes of producing engineers who are creative, innovative and adaptive with skill sets in line with the Engineer of 2020 attributes. Insight from participants and examples of their efforts are typical tools CIAN uses to measure the achievement of learning outcomes.
Some amount of change is expected from engagement with each of the main CIAN activities; however, students who participate the most in CIAN’s education activities will likely see the biggest achievement of learning outcomes. These students tend to be our Student Leadership Council (SLC) officers and representatives. Tracking these students post-graduation is important. The SLC represents the nexus of the ideal research, industry, and education overlap of goals and opportunities. The long-term intent for assessment of CIAN graduates will include all students (undergraduates and graduates) and post docs active in CIAN research labs and enrolled in CIAN-related courses. However, the focal point of the longitudinal assessment will be the members of the SLC.
The assessment tools used to measure the outcomes of each activity are included in the method to assure proper program management.
Impact/benefits: Program evaluations provide valuable feedback from participants, enabling CIAN to modify its education programs each year and continually improve outcomes (e.g., immediate jobs in industry for CIAN graduates, REUs prepared for graduate school, or K-12 students interested in pursuing education in optics).
Sustainability: With the help of a graduate assistant working specifically on assessment and evaluation, CIAN’s education director has clarified the goals of CIAN education assessment and begun a targeted effort to improve assessment strategies, develop new assessment tools, and obtain more quantifiable data going forward. These assessment methods require minimal modification each year, thus lending to the ease of this assessment plan’s sustainability.
Tips:
Collaboration with more established ERCs is recommended in order to exchange ideas regarding measurement tools and assessment plans; likelihood of success is increased when building on others’ time-tested methods.
It is best practice to stagger the distribution of surveys and other self-report tools several months apart to avoid “burn-out” and low response rates from the same pool of respondents. To maximize response rates, make sure to send out one or two reminder emails, along with a deadline, following the initial invitation to participate in surveys.
Preparation of the assessment tools should begin long before the first day of programs so there is sufficient time to address “hiccups,” unforeseen obstacles, and make edits to surveys and other tools.
1.3. Center: ERC for Revolutionizing Metallic Biomaterials (ERC-RMB)
Lead Institution: NC A&T State University
Center Director: Dr. Jagannathan Sankar, Department of Mechanical Engineering
Name of Program: Educational Activities and Assessment
Type of Program: Assessment examples of Research Experience for Undergraduates (REU) / Research Experience for Teachers (RET) / Young Scholars (YS) Programs
Program Synopsis: The ERC-RMB has supported REU/RET/Young Scholars summer programs at North Carolina Agricultural and Technical State University. These programs link ERC-RMB researchers/scientists, North Carolina A&T faculty, and graduate students with undergraduate, teacher, and young scholar participants in teaching and learning activities for six weeks. REUs have come from the ERC-RMB lead institution North Carolina A&T, partnering institutions (University of Cincinnati, University of Pittsburgh), and from universities from across the country. RETs come from K-14 environments, and Young Scholars were middle and high school students, from the local school district. During the six-week program, REU/RET/YS participants engaged in laboratory work, classroom instruction, seminar discussion, and field trips to local industry in bioengineering. Participants had research responsibilities carried out under the guidance of the ERC-RMB mentors (i.e., research/scientists). REU/RET/YS participants completed pre-/post-assessment surveys to determine changes in their understanding related to bioengineering, creativity and innovation, diversity of thinking, entrepreneurship, and ethics. RETs were assessed for their perceptions of programmatic activities upon K-14 teaching and learning. REU/RET/YS participants engaged in weekly focus groups, and assessment personnel gathered observational data from laboratory activities.
Contact person/website: Robin Guill Liles, Associate Director for Educational Assessment (rgliles@ncat.edu), http://erc.ncat.edu/
Dates of Operation/Timeframe: The REU/RET/YS program associated with the ERC-RMB has been operational each summer since 2009. The program is designed to last at least six weeks. REU/RET/YS program is a residential camp whereby participants live on the North Carolina A&T campus. Activities were planned from 8:00am–9:00pm.
Background: Initially, the REU/RET/YS program served as the gateway educational activity for the ERC-RMB Education and Outreach (E&O) program. This program was included in the original ERC-RMB E&O conceptual plan. Over the years, the REU/RET/YS program has continued to grow and diversify. Faculty and research/scientist mentors have refined their teaching modules, both within and out of the laboratory. Field trips and other extracurricular activities have become more sophisticated. Most notably, framers of the ERC-RMB E&O programs continue to recruit and retain record numbers of participants from underrepresented populations.
Assessment Methodology: In the first year of the Center’s REU/RET/YS program, the assessment team implemented pre-/post assessment interviews. Interview questions flowed from the ERC-RMB E&O mission statement to “to train future engineers for industry, research, and development in a multidisciplinary environment that values diversity of thinking, creativity and innovation, and entrepreneurship.” Interview transcripts revealed frequently occurring themes and factors. In year two, a written pre-/post- assessment instrument was developed focusing upon these themes and factors. The 80-item survey was administered electronically through Survey Monkey. In addition, the assessment team initiated a series of weekly focus groups. In an effort to refine assessment data, satisfaction questionnaires were implemented in the third year of the REU/RET/YS program. Their purpose was to determine which extracurricular activities participants deemed most beneficial to their REU/RET/YS experiences. From the first three years of assessment data, it became evident that REU/RET/YS participants found laboratory experiences important to their positive change in understanding and knowledge related to bioengineering. For this reason, in the fourth year, the Home Observation Scale (HOME) was adapted to use in a series of laboratory observations. Two members of the assessment team systematically observed REU/RET/YS participants as participants engaged in their laboratory activities. Once observations were completed, team members compared observations. Only in those observations with 100% observer rate agreement were reported. In year five of the ERC-RMB REU/RET/YS program, a concert of assessment instruments were used, including pre-/post survey assessment, focus groups, satisfaction questionnaires, and laboratory observations.
Impact/benefits: Assessment data from the REU/RET/YS program provide impressive evidence that a portion of participants go on to self-select into academic and career bioengineering strands. From the ERC-RMB ethical perspective, this selection process has particular value in bringing members of under-represented populations into the field of bioengineering.
Tips: Developing assessment strategies for the first iteration of educational and outreach programs and activities can be a time-consuming and even overwhelming process. Many young ERC E&O personnel report confusion about the timeline and impending site visit. Having a professional with particular expertise in assessment is helpful. In addition, good note- and record-keeping is a must for writing cogent end-of-year reports.
2. Education Assessment Research
2.1. Center: ERC for Quantum Energy and Sustainable Solar Technologies (QESST)
Name of Program: Conceptions of Engineering Scale
Type of Program: Precollege engineering education research project
Program Synopsis: This project addressed an engineering education knowledge gap through the development of a valid and reliable instrument to measure middle school students’ conception of engineering based on analysis of three conceptual frameworks currently used in K-12 engineering education.
Contact person/website: Michelle Jordan (michelle.e.jordan@asu.edu), Chrissy Foster (christina.foster@asu.edu), Jenefer Husman (jenefer.husman@asu.edu)
Dates of Operation/Timeframe: 2013 & continuing
Background: Educational research results suggest exposure to engineering needs to begin in middle school in order to provide equitable opportunities for students to consider majoring in STEM fields. However, students’ conceptions of engineering may be inaccurate, resulting in dropping out of engineering degree programs. It is important to understand students’ early conceptions of engineering and how educational experiences influence those conceptions. Although inroads have been made in identifying developmentally appropriate knowledge and skills related to engineering, there is a need for a conceptually sophisticated understanding of engineering knowledge to guide K-12 standards and assessment. To date, few researchers have attempted to measure middle school students’ conceptions of engineering.
The student sample targeted for the study is of particular interest as they are representative of populations currently under-represented in STEM careers (Gibbons, 2009) and often under-prepared for STEM careers by K-12 schools (Museus et al, 2011).
Note: This project was funded with a separate grant focused on engineering education research, not from ERC core funds. The ERCs are not expected to conduct engineering education research with core funds but rather, hopefully, to use the outcomes and results of prior or parallel research to help improve the execution, monitoring, and improvement of their programs.
Methodology: Assessment Scale Development
Six broad aspects important to middle school students’ conceptions of engineering were identified:
The purpose of engineering to create a solution to a problem that exists at multiple levels of society
The nature of engineering as a systematic, iterative, and creative process with requirements and constraints
Phases/elements of the design process
Engineers use certain kinds of tools in particular ways for specific purposes
Engineering is a highly social activity requiring collaboration and communication with diverse people for multiple purposes using a variety of mediums
Engineers develop habits of mind (e.g., optimism, creativity, systemic thinking)
Having identified these six broad aspects of engineering education, they were considered in relation to common misconceptions of engineering previously identified. Additional items were constructed that asked respondents to reflect on structures, behaviors, and functions of engineering. Initial items were field-tested using interviews with practicing engineers. Items were added, deleted, and revised based on their feedback. These activities resulted in the development of an initial 25-item multiple-choice instrument comprised of six subscales to measure students’ conceptions of engineering.
Notes: Scale development was based in part on findings from 2011-2012 data collected in nine Math Engineering Science Achievement (MESA) Clubs. Study participants were recruited from middle school students who participated in afterschool engineering clubs during 2013-2014. Likert-items, open-ended survey questions, and interviews yielded insights into middle school students’ conceptions/ misconceptions of engineering, self-perception of themselves as engineers, and perceptions of their MESA experiences. Analysis suggested that students valued their learning from hands-on collaborative challenges. However, they varied considerably in the sophistication of their understanding of what engineering is and what engineers do (Jordan, Foster & Husman, 2013a, 2013b).
Impact/benefits: The Conceptions of Engineering Scale was field tested by being administered to 100 participants diverse in age (sixth through eighth graders), ethnicity, gender, and socio-economic status (SES). Fifty-nine respondents had participated in an engineering after-school club (MESA); 41 respondents had not participated. MESA is a co-curricular afterschool program, currently offered in eight states. MESA clubs are typically advised by volunteer teachers. MESA traditionally supports educationally disadvantaged students in Title I schools by providing pathways for minority, low-income, and first generation college-bound students to succeed in STEM disciplines. Students collaborate in design challenges and compete in statewide completions. Results are currently being analyzed.
Interviews were conducted to probe students’ reasoning related to their responses. Using think-aloud protocols, we asked students to explain their answers to three to six survey items (e.g., “What were you thinking when you responded to this question?” and “Why did you select that response?”).
The development of a valid and reliable instrument to measure middle school students’ conceptions of engineering can be used to a) design and evaluate the impact of in-school and after-school programs that emphasize collaborative project-based, engineering design learning experience, b) improve outreach programs which have as their goal to improve knowledge and understanding of engineering and promoting engagement in STEM, c) influence research about STEM activities and coursework as it relates to the recruitment and retention of middle school students’ from diverse backgrounds, and d) identify new information about students’ existing knowledge and understanding of engineering that middle school science and math teachers will find useful for assessing conceptions of engineering as they begin to implement the Next Generation Science Standards.
Tip: Further scale development is planned. Pre- and post-administration of the scale to a group of 600+ middle school members of MESA clubs is planned for 2013-2014.