Accreditation

Standard R4 Program Impact

Introduction

The Educator Preparation Program (EPP) at the University of Alaska Southeast investigated the degree to which program Completers: effectively contribute to P-12 student-learning growth (R4.1a); and successfully apply the professional knowledge, skills and dispositions in a manner that the teacher education experiences were designed to achieve in P-12 classrooms (R4.1b) through a case study of recent Completer effectiveness and the administration of relevant surveys of recent Completers (Alumni) and Employers.  A comparison of student achievement (MAPS standardized test) between the students (3^rd – 9^th grades) of the UAS cohort of teachers (Case Study participants) and the over-all scores of Juneau and Anchorage students is used to address R4.1a; and the classroom observations conducted as a part of the Case Study, including STOT evaluation scores, is used to address R4.1b.  Findings from the Case Study conducted with recent Completers, and an analysis of the Alumni and Employer surveys, are presented in this report. The details of the Case Study methods are presented in a separate overview of the Case Study methodology. 

The EPP addressed the question of Employer satisfaction with the Completers’ teacher education preparation for their assigned responsibilities in working with diverse P-12 students and their families (R4.2) through the administration of an Employer survey. The evidence used to examine this component of the program impact comes from three data cycles of the Network for Excellence in Teaching (NeXT) Employer survey (2021, 2022, and 2023). 

Finally, the EPP investigated the degree to which program Completers perceive their teacher education experiences as being relevant and effective in preparing them to successfully meet the professional challenges and responsibilities which they face in their P-12 classroom assignment (R4.3).  To address R4.3 the EPP collected and analyzed data from three cycles of NeXT surveys of Completers (2021, 2022, and 2023) and analyzed the Case Study data collected during the focus group discussions and individual teacher interviews.  Additionally, the EPP includes the rationale for the data selection and the analysis procedures employed for each R4 program impact component in the attached methodology overview.  

Surveys Note: The NeXT Surveys (alumni and employer)

These surveys have been developed and validated through North Dakota State University, and administered through North Dakota State University for the 2021 and 2022 data cycles.  Beginning in the fall of 2022, UAS took over the administration of these surveys, with the goal of increasing response rates.  The UAS administered surveys in the 2022 – 2023 data cycle are identical to the surveys previously administered by North Dakota State University.

R4.1: Program Effectiveness and the Multi-Case Study of Graduates

The provider demonstrates that program completers effectively contribute to P-12 student-learning growth (R4.1a); and apply in P-12 classrooms the professional knowledge, skills, and dispositions that the preparation experiences were designed to achieve (R4.1b).

           To collect measures of Completer effectiveness and program effectiveness the EPP used a Case Study approach (see R4-Case Study Methodology). The Case Study included a comparison of the achievement scores of students of program Completers who participated in the study with overall student achievement data from both the Juneau and Anchorage school districts. The comparison was used to measure teacher impact on P-12 student learning (R4.1a).  Additionally, half day observations using the EPP’s classroom teaching evaluation instrument: Student Teacher Observation Tool (STOT), was used to measure teacher effectiveness (R4.1b).

The first phase of the Case Study of Graduates was conducted in 2021through the use of classroom observations and focus group discussions with 10 Juneau School District (recent Completers in years 1 – 3 of teaching) volunteer participants. These same data collecting activities were repeated in 2022 with volunteer recent Completer participants from the Anchorage School District.  In 2023 a series of eight individual interviews were conducted with volunteer recent Completers teaching in various rural school districts serving primarily Alaska Native communities. Each iteration of the multi–Case Study project included graduates from each of the initial licensure programs: Elementary bachelors, Elementary MAT, Special Education bachelors and MAT, and Secondary MAT.

Goals of Case Study

The goals of the Case study were to: 1) determine the impact and effectiveness of UAS initial licensure teacher preparation program through observations of and interviews with recent Completers in three distinct professional contexts (urban, medium sized community, and rural); 2) measure the impact of our graduates’ teaching on student achievement scores;  3) observe graduates in practice relative to Interstate Teacher Support and Consortium (InTASC)  core teaching practices;  4) triangulate data collected through NeXT survey instruments with observational data and interview and focus group discussion data from the 28 study participants; 5) better understand the types of supports that rural teachers need to enact culturally sustaining pedagogical practices when teaching in predominantly Alaska Native communities; and 6) make changes to the initial licensure programs based on continuous improvement recommendations ensuing from the analysis of gathered data.

Participant Selection Criteria.

Participants for the 2021, 2022, and 2023 multi case study were recruited from graduates in each of the licensure areas who were “early career” (years 1 – 3) teachers. The participants were selected, in part, to attempt to achieve a representative sample in terms of gender, ethnicity and whether they completed an undergraduate or post-baccalaureate certification program.  In 2021, there were 10 participants with 5 from Elementary, 3 from Secondary, and 2 from Special Education. In 2022, there were ten participants with 6 from Elementary and two from Secondary and two from Special Education programs. For the 2023 Case Study, participants included three Indigenous and five Caucasian graduates from UAS initial licensure programs teaching in various rural, predominantly Alaskan Native school districts. Each of the three Case Study cohorts included female and male participants, as well as both racial and ethnic majority and minority identifying participants. 

Data sources for the first two iterations of the Case Study (Juneau and Anchorage teachers) included focus group discussions and classroom observations with the STOT.  The third iteration of the Case Study drew on one-to-one interview data with the 8 rural Alaska teachers. The classroom observations were conducted using the Skills of Teaching Observation Tool (STOT) which is the same validated instrument that was used during the internship / student teaching phase of the participants’ pre-service preparation program. The STOT evaluations measured teaching effectiveness in the domains of Learner and Learning, Content Knowledge, Instructional Practice, and Professionalism using a 4-point scale of: distinguished (4), proficient (3), emerging (2), and underdeveloped (1). The STOT observation scores were used as evidence to measure Completer teaching effectiveness (R4.1b).  The focus group discussion data and rural teacher interview data were used to assess Completer satisfaction (R4.3) and is discussed in that section of this report.

R 4.1.a and b.: Evidence of Program(s) Effectiveness

The Case Study of Graduates, the NeXT Surveys, and comparative student performance data from the Juneau and Anchorage school districts, provided the EPP with valuable feedback on its impact and the satisfaction of both its completers and their Employers.  Both strengths and program deficiencies were revealed through the analysis of data collected across the data sources. While program deficiencies were noted and plans to address these deficiencies were incorporated into EPP-wide and program-level continuous improvement review efforts (annual updates of preliminary case study findings have been provided to faculty and to our external advisory committee in 2021, 2022, and 2023), the evidence from the data supports the broad finding that Alumni have had a positive impact on P-12 student learning and that both the Completers and their Employers evinced an over-all high degree of satisfaction with the preparation they received.  

Observations with STOT

The analysis of the evidence derived from the observations demonstrates that the preparation delivered by the EPP has a positive impact on the effectiveness of its Completers. In the summary of findings, the 2021 case study found from the STOT observational evaluation scores that 100% of graduate participants observed were rated as being proficient to highly proficient or distinguished in all areas. The 2022 case study also found that 100% of its participants were proficient to highly proficient or distinguished in all ten of the InTASC standards rated through the STOT observation evaluation instrument.

Additionally, data from the on-site evaluations conducted during the Case Study provide evidence that the participants apply the professional knowledge, skills, and dispositions corresponding with teaching effectiveness. Participant pedagogical content knowledge was measured on the STOT in the domain: Content Knowledge, on a scale of 1-Underdeveloped to 4-Distinguished.  Participants scored an average of 3.8 in both 2021 and 2022. The two domains on the STOT evaluation that measure participants’ effectiveness with their learners are Learners, Learning, and Diversity and Instructional Practice. The mean scores for participants in these two domains were 3.9 in 2021 and 3.8 in 2022 for Learners, Learning and Diversity, and 3.8 in both 2021 and 2022 in Instructional Practice. Finally, Professional Learning, Ethical Practice, and Leadership and Collaboration were measured in the domain: Professionalism, and the mean score for participants was 3.9 in 2021 and 3.8 in 2022. The scores across each of the domains measured in the STOT demonstrate a high level of participant proficiency in the application of the necessary knowledge, skills, and dispositions to be an effective educator. 

Moreover, the comparative student achievement data collected on UAS Alumni early career teachers (1 – 3 years) for both the Juneau (N = 15) and Anchorage (N = 17) teachers showed a favorable statistical comparison with the overall district averages.  The “Alaska Star” MAPS 2022 fall and 2023 spring test results (grades 3 – 9) were used to make this comparison. For the Juneau cohort, the math and reading scores were 8% and 10% higher, respectively, for the students of the UAS cohort of teachers, compared to the overall district averages.  For the Anchorage cohort, the averages were identical for math and 2% lower for reading on the Spring 2023 test results.  Of note is that 12 of the 17 Anchorage cohort teachers are teaching in Title I schools, in a district that has only 1/3 Title I schools; the adjusted student performance results show that the UAS cohort’s students scored approximately 5% higher than the district average for a demographically similar comparison group.  In both cases the teachers in the UAS cohorts are in their first through third year of teaching.  This factor of having less experience than the average teacher in both districts is further evidence of the effectiveness of the UAS teacher preparation program.

R4.2: Employer Satisfaction and the NeXT Supervisor Survey (SS)

The provider demonstrates that employers are satisfied with completers’ preparation for their assigned responsibilities.

The Employers of the Alumni of the EPP’s initial licensure programs (Bachelors in Elementary, Masters in Elementary, Masters in Secondary, and Masters and Bachelor’s degrees in Special Education) are asked to participate in the Supervisor Survey (SS) also developed by the Network for Excellence in Teaching (NeXT).  This survey is sent to the supervising principals of first year alumni in the spring of their first year of teaching after completing their respective preparation program.  The survey questions probe the supervising principals on their perceptions of the novice teachers’ performance in several necessary knowledge, skill, and disposition domains related to their teaching responsibilities. The SS aligns with the Interstate Teacher Assessment and Support Consortium (InTASC) standards with survey questions developed to measure satisfaction in four main areas: Instructional Practice, Diverse Learners, Learning Environments, and Professionalism. 

The response rates for the NeXT SS have been above 40% for the last three cycles of data. Specifically, the response rates for the EPP were 40% in 2021, 46% in 2022, and 37% in 2023. The survey responses include a representative sample from the elementary and secondary programs. (See 2021-2023 NeXT Employer Survey Data in R4.2 Employer Satisfaction)

Data collected from the case study was consolidated with the data collected from the NeXT surveys of alumni and employers. These surveys are described in the following sections. To view the suite of NeXT surveys and their validity and reliability please see R5.2 NeXT Surveys with Reliability and Validity.  All data from the NeXT surveys have been consolidated into “across the EPP totals,” and presented through program specific disaggregation tables.  In the following sections, you will find reference specifically to data from the Transition to Teaching and Supervisor surveys. 

R4.2 Evidence for Satisfaction of Employers

An analysis of the data from the NeXT Supervisor Survey shows that employers report that Completers teaching in their schools are effective practitioners. When asked to what extent they agree or disagree that the Completer effectively teaches the subject matter in his/her area, there was an average of 97% agreement in the last three years (100% in 2021, 100% in 2022, and 90% in 2023). With respect to effectively teaching students from culturally and ethnically diverse backgrounds and communities, the average percent agreement was 95% (100% in 2021, 96% in 2022, and 90% in 2023).  Relatedly, there was an average percent agreement of 98% (100% in 2021, 100% in 2022, and 96% in 2023) that Completers effectively connect core content to students’ real-life experiences. Additional areas in which employers report a high average percent agreement across the last three years with regard to the effectiveness of completers teaching in their schools include the following: using Digital and Interactive Technologies to achieve learning goals (98%); Differentiates Instruction for a variety of learning needs (95%); and creates a Learning Environment in which differences such as race, culture, gender, sexual orientation, and language are respected (93%). 

Supervisors (principals) completing the Supervisor Survey shared such comments as: “. . . has been an invaluable asset to our school . . . has adjusted very quickly to the pace and information load many first-year teachers struggle with . . .professional, prepared, reliable, and an excellent communicator with students and parents . . . performance thus far is what I would expect from a teacher with much more experience.”  Overall, across the entire EPP, for initial licensure programs, the respondents to the Supervisor Survey average aggregate scores on all items, on a 1 – 4 scale ranging from 3.2 for Classroom Management to 3.8 for “effectively teaches the Subject Matter in my licensure area.”  Other items with high average scores (above 3.6) included: high Learning Engagement levels, makes Interdisciplinary Connections, creates a respectful Learning Environment for differences in gender, race, ethnicity, and sexual orientation.

R4.3: Satisfaction of Completers and the NeXT Transition to Teaching Survey (TTS)

R4.3 The provider demonstrates that advanced program completers perceive their preparation as relevant to the responsibilities they confront on the job and that the preparation was effective.

            Alumni of the EPP’s initial licensure programs are asked to participate in the Transition to Teaching Survey (TTS) developed by the Network for Excellence in Teaching (NeXT). The survey is sent to first-year alumni in the spring of their first year of teaching after completing their respective preparation program. The survey questions probe alumni’s employment status, their processes for finding employment, and their level of satisfaction with their preparation program in preparing them for the necessary knowledge, skills, and dispositions related to their teaching positions. The TTS aligns with the Interstate Teacher Assessment and Support Consortium (InTASC) standards with survey questions developed to measure satisfaction in four main areas of instructional practice, diverse learners, learning environments, and professionalism. 

The response rates for the NeXT TTS have averaged 20% for the last three cycles of data. Specifically, the response rates for the EPP were 18% in 2021, 20% in 2022, and 22% in 2023. The survey responses include a representative sample from the Elementary and Secondary General and Special Education programs. (See 2021-2023 NeXT Alumni Survey Data in R4.3 Completer Satisfaction).  

Special Note: Covid Consideration

In the discussions that follow, the observation data that was collected in 2022 aligns with both the 2021 and 2022 cohorts whose teaching circumstances for the 2020 – 2021 school year was adapted to COVID related contextual considerations (i.e.., online-learning, hybrid learning, school closures, etc.). This delay in observation data collection for the first cohort (Juneau teachers) afforded the EPP an opportunity to make observations of classroom instruction and to use the STOT observation and evaluation protocols in similar manner for both cohorts.  Likewise, the memorandum of agreement with the Juneau School District was extended through the 2023 school year so that student performance data could be collected at that time (spring of 2023) for both cohorts, rather than during the COVID disrupted school year of 2020-2021.

R4.3Evidence of Satisfaction of Completers

The NeXT Transition to Teaching and Supervisor Surveys of alumni and employers along with qualitative data from the case studies of graduates provide evidence that the EPP’s completers and their employers are satisfied with the relevance and effectiveness of the preparation it provided.

The data from the NeXT Transition to Teaching Survey shows that the majority of responding Alumni from the last three years who have gone into teaching report satisfaction that the preparation they received was relevant to their responsibilities as novice teachers. Across the last three years, an overall average of 92% of responding Alumni agree that they would recommend their teacher preparation program to a prospective teacher (100% in 2021, 90% in 2022, and 86% in 2023). Additionally, an overall 90% agree that their teacher education program prepared them to be successful in their current teaching position (94% in 2021, 90% in 2022, and 86% in 2023). 

In examining each of the surveyed skills, a majority of responding Alumni report that they felt prepared for various skills in instructional practice, learning environments, and professionalism. There was over 80% agreement of preparation for all but one instructional practice skill from the responding AY2021 cohort, all but three skills from the responding AY cohort, and all but two from AY2022. For skills related to Learning Environments, the responding AY2021 cohort had 80% agreement they were prepared in all but two skills, and the AY2022 cohort had only one rated below 80% agreement. The AY2021 cohort who spent almost the entire internship year in a virtual or in-flux Learning Environment, had above 80% agreement at feeling prepared in half of the learning environment skills. For professional skills, over 80% of responding completers across cohorts felt prepared in all but two of the surveyed skills.

Focus Group Discussions

Comments from the participants during the focus group interviews suggest that Completers identify that they are having a positive impact on their students' learning. Participants shared: “I was prepared and ready to teach when I graduated; the yearlong internship taught me to pace, assess in the moment, and handle classroom behaviors,” (BA Elementary).  “I gained the foundation needed to become a successful teacher; the UAS MAT program was a lot of hard work, but teaching is hard work as a profession and the program prepared me for that.” (MAT Elementary).  “. . . great preparation because so much of the program was actually spent in the classroom; talking with my cohort of classmates and professors about what was and wasn’t working was extremely valuable,” (MAT Secondary). “I felt that my special education training was great in getting me ready for my first year; I loved my time at UAS.  I know that it’s impossible to teach us and offer us everything we need to know our first year, but I’m very appreciative of the research-based teaching strategies and resources made available to us,” (BA Special Education). 

Program(s) Strengths and Deficiencies: Thematic Findings from the Triangulation of the Case Study and Survey Data

Strengths:

• Relationships between faculty and students and among students in student cohort groups was the most frequently cited strength of the collective EPP programs, using both interview and survey data.  Students repeated cited their appreciation for the individual attention, flexibility, and support of faculty, and the high degree to which they benefited from their student cohort groups throughout the duration of their teacher preparation programs.
• Rigor of program content and field experience (practicums and internships) was the second most cited strength of the collective program designs across the EPP.
• Relevance of course content and practicum coursework assignments, including the cross program emphasis on culturally responsive pedagogies and “real world,” place based rooted curricular design, was the third most cited strength of each of the EPP’s initial licensure program.

Deficiencies:

• Lack of information and preparation regarding “trauma-informed” pedagogical practices was the most frequently cited “gap” in our Completers’ teaching preparation.
• Lack of specific program design support to accommodate the growing number of student interns and student teachers experiencing their internships while also serving as the teacher of record (i.e. student teaching in their own classrooms) was the second most cited area of need by our recent Completers, from both the survey and interview data.
• Lack, at times, of inter-program coordination and collaboration between general and special education teacher programs, leading to students in both programs less than fully prepared to collaborate on overlapping professional responsibilities such as the creation of Individual Education Plans, (IEPs) was the third most cited area of deficiency.

Conclusion

The analysis of the data gathered from the Case Study of graduates, the Next Alumni Survey (Transition to Teaching Survey), and the NeXT Supervisor Survey indicates that the EPP’s Completers advance the learning and development of their students through ample content knowledge, effective instruction that accounts for individual differences, and a stance that is respectful and inclusive of diverse students (R4.1).  This analysis further demonstrates that the Employers of UAS Alumni are satisfied with the Completers’ professional preparation vis à vis their teaching responsibilities (R4.2).  Finally, the analysis of this data shows that the recent Completers perceive their professional preparation as being both relevant to and adequate for effectively meeting their teaching responsibilities (R4.3).

UAS Employer Survey 2023 (Initial Licensure Programs Only)

Employers agree that first year teachers do the following:

Initial Programs - Praxis II

Elementary Education

Content Knowledge (5018/00018), Highest Score

Curr Instruction & Assessment (5017/0017), Highest Score

Secondary Education

English Lang Arts: Content Knowledge (5038/0038), Highest Score

Social Studies: Content Knowledge (5081/0081), Highest Score

World and US History: Content Knowledge (5941/0941), Highest Score

Special Education

Core Knowledge & Applic (5354/0354), Highest Score

Source: Educational Testing Service

Advanced Programs

Portfolio Pass Rates by Program

Completers* Working in Alaska

*A completer is defined as a candidate who successfully satisfied all program requirements of a teacher preparation program at least six months previously and who is currently employed in a position for which they were prepared for state licensure.

Graduates working in Alaska in AY2023-24 in positions they were not prepared for are not included in these calculations. 

Source: UA Decision Support Database & State of Alaska Dept of Education Employment Statistics