Faculty and Professional Learning Community Program Assessment: 2006-7
From CTLpedia
Brad Kincaid, FPLC Director and Life Science Faculty, Mesa Community College
This assessment has been submitted to the Learning Communities Journal. You can access this paper here as a pdf document. The data are the same as below, however the paper expands on the potential of the program to transform us into a more learning-centered college.
Contents |
Summary
The FPLC Program goals are connecting and engaging faculty and staff to improve teaching and learning. Interdisciplinary groups engage in seminars, retreats, and communication about a selected topic. This program has engendered an unprecedented level of engagement by over 70 participants that has reinvigorated members and bridged disciplinary boundaries. Assessment results indicated that we were signifiantly successful in achieving our goals of connecting faculty and staff and engaging faculty and staff in enhancing teaching and learning.
Program Description
Our Faculty and Professional Learning Communities (FPLC) Program is integral to helping MCC strive for and reflect its values of learning, excellence, inclusiveness, and community. Our FPLC Program is an institutional process promoting collaboration, communication and scholarship to achieve our strategic goals: excellence, access, student support, inclusiveness, organizational effectiveness, professional development, community engagement, and effective technology use.
Our goals and objectives include:
- Promote connections among faculty, staff and students at MCC.
- Build college wide community
- Increase interdisciplinary collaboration
- Encourage coherence of learning across disciplines
- Investigate and incorporate diversity
- Promote reflection and engagement in improving teaching and learning at MCC.
- Support teaching and learning initiatives
- Broaden assessment of learning
- Increase focus on learning
- Reward and recognize excellence in teaching
- Nourish the scholarship of teaching and learning (SOTL)
As defined by Milt Cox at Miami University (Ohio), an FPLC is an interdisciplinary group of faculty and staff who engage in an active collaboration focused on enhancing teaching and learning, which is broadly defined to include anything that can improve student success. FPLCs ideally include 8-12 members who typically meet biweekly with a facilitator for at least a year. FPLCs engage in self-selected activities that promote learning, professional development, community building, scholarly teaching, and the scholarship of teaching and learning. Participants usually select a focus course or project in which to implement an innovation related to the FPLC topic and collaborate to assess impacts on teaching and learning. Local communication about FPLC outcomes is a critical component of the program leading to enhanced campus community and improved teaching and learning.
As a tool for creating community and engaging members in pursuing our objectives, our 2006-7 FPLC Program supported six new topic-based communities: Sustainability, Undergraduate Research, Nature of Science, Humanities, Global Learning, and Podcasting. Over 70 faculty and staff members applied and were accepted into the program.
FPLC Member Expectations
Active, reliable, supportive, and contributing membership
FPLC members are expected to be active, reliable, supportive, and contributing members of the FPLC. We expect members to be flexible in scheduling and attending community events outside normal work hours if necessary. They must commit to attending seminars comprising 4-6 hours per month, to help build a safe support group for all participants, and to explore and discuss the literature related to their discipline and the FPLC topic.
Evaluation project
FPLC members are encouraged and supported to conduct an evaluation project related to the FPLC topic. We expect all members to focus on an idea emerging from the literature of the FPLC topic and encourage all members to select a focus course or other college project for implementation. We also encourage and support the assessment of the implementation of their idea by providing classroom research consultation.
Communicate findings
FPLC members are expected to communicate findings of the FPLC and their reflection on teaching and learning. We expect members to share literature reviews related to the FPLC. They must document their reflection on teaching and learning that led them to their idea. We support them in documenting their implementation and assessment of their idea. We are making FPLC documentation public to the local community by publishing it on the CTL website via the CTLpedia and by hosting a campus-wide teaching and learning conference scheduled for August 17, 2007. Support is available for those who wish to present their work at a regional or national conference or those who wish to publish their work in a multidisciplinary or disciplinary journal. We hope that some will consider continuing their work with a campus or district grant or a sabbatical.
Program Assessment Methods
The primary means of assessing our FPLC program was a survey adapted from Milt Cox's FLC survey (components & professional development, student learning and results). The survey was administered in a pre/post participation design using an online survey tool licensed from SurveyShare.com. There were three areas assessed by the survey: program components, participant professional development, and student learning impacts.
The schedule for 2006-7 had to be adjusted to account for some delays in getting the program authorized during the spring of 2006. Consequently, recruiting did not begin in earnest until May and therefore it was necessary to allow for a short fall recruiting window, which was open until the Opening Retreat at the end of September. To generate additional interest, an FPLC workshop (also presented at AACU and PKAL) was held on August 16, 2006 just before school started.
Common Events for 2006-7
- FPLC workshop: 8/16/06
- Opening Retreat: 9/30/06
- Midyear Retreat: 1/9/07
- Yearend Retreat: 5/11/07
- Mesa Conference: 8/17/07
One consequence of the delayed schedule was that pre-participation survey was open for most of the month of October after many FPLCs had begun to meet. Thus, there is a potential that the pre-participation responses were affected by early experiences with the FPLCs. Furthermore, the FPLC workshop may have raised expectations regarding some program outcomes.
For the program components and professional development, participants were asked to rate impacts on a 10 point scale from very little to very great impact. For the student learning impacts, participants were asked to rate student learning impacts on a 5-point scale: students learned less, no change, or learned more to a small, medium, or great extent. In addition, there were many opportunities for open text responses. Specific questions are included with the results below.
Not all members who took the post-participation survey took the pre-participation survey. Consequently, the pre/post comparison is limited to the 43 matched pre/post surveys. Skipped questions and “Not Applicable” responses further reduced sample size for some questions.
Participant responses should be examined both for differences among items within the program components, professional development and student learning question categories to estimate the relative importance of each item and between pre/post responses to determine if the experience had a significant impact on the perceived importance of the items. Error bars (2 SE) on the graphs indicate the approximate significance of the differences.
Among FPLC differences were analyzed in two ways. First, pre/post difference scores for each survey item were analyzed in an among FPLC ANOVA. This approach is equivalent to a covariance analysis with a highly correlated covariate. In addition, post-participation responses were considered alone in an among FPLC ANOVA. Not only did this increase the sample size for some groups, but it also eliminates the possibility that meaningful information was lost in the covariate. The latter may occur when the pre-participation responses include some of the group effect. The may have been the case because the pre-participation covariate may have reflected some raised expectations developed during the FPLC workshop and by early experiences in their FPLC.
Program Assessment Results and Discussion
Seventy-two people applied and were accepted into the FPLC program for 2006-7. Of these, only four members of the Nature of Science FPLC were continuing members; the rest were new to the FPLC program. Notice that two of the FPLCs had co-facilitators. 83% of the members were fulltime faculty with a few other employee groups and students. The students did not persist in their FPLCs for various reasons.
Of the high point (HP) members, 55 took the pre-participation survey even though the date for completion was extended until the end of October. The active membership (Act) was determined by the members listed midyar by the facilitators on their FPLC group page. By the middle of July, 46 members had completed the post-participation survey. Retention and attrition may reflect some aspects of the FPLCs, but can have multiple causes. The attrition from high point to active ranged from zero to 25%. Survey completionrates may relfect levels of engagement and connectedness to each FPLC and the overall program. The low sample size for some FPLCs may limit the meaninfulness of some among FPLC differences in responses.
Impacts of FPLC Program Components
The first set of questions asked members to estimate the impact on them of the major program components. Colleagueship and facilitators were by far the two highest rated components of the FPLC program with average post participation scores of almost 9 on the 1-10 rating scale. Four other components (retreats, seminars, projects and resources) were clustered just above 8. Two components of the program were rated quite low: the planned conference and the CTLpedia document database.
Table of pre/post t-tests…
Pre/post paired comparisons in ratings of program components revealed only two significant differences. The ratings of the facilitators increased significantly in the post-participation survey, and the rating of the teaching project decreased significantly in the perceived impact on the participants. The conference and CTLpedia questions were not included in the pre-participation survey. The fact that these two components were added late to the program probably accounts for their low ratings.
None of the pre/post differences were significant among FPLCs at the 5% significance level. However, the ratings for the Seminars/meetings was nearly significant with p < 0.09. When only post-participation scores were considered, seminars/meetings among FPLC difference became significant at the 0.05. In addition, the among FPLC rating of the CTLpedia was nearly significant (p < 0.07). The ratings of retreats/conferences indicate some differences, but the low sample size makes this difference questionable.
A plot of the post participation means of the seminar/meetings revealed that NOS and UGR FPLCs rated their meetings very highly, whereas the ratings of the others were clustered below a score of 8.
The among FPLC plot for CTLpedia shows the relatively low rating of this program component for all FPLCs with an exceptionally low rating by the Sustainability FPLC.
Discussion of program components.
Colleagueship was highly rated in the pilot program too. This may be the most significant aspect of the program. It corresponds to our goal of connecting faculty and staff and building social capital. Lack of connectedness is symptomatic of the isolation of our departments and in society.
Comments about Colleagueship…
The high rating for facilitators suggests their importance to the functioning of an FPLC. It is especially significant that this rating increased significantly between the pre- and post surveys. This also supports our focus and investment in facilitators.
Comments about facilitators…
The significant drop in rating of the project was disappointing. This should be a learning experience for the participants. However, compare this result with that below for the effect of the project on student learning hwere the result was just the opposite.
The low ratings of the conference and CTLpedia were also disappointing, but, given that the conference had not occurred and the CTLpedia was a new addition to the program during the spring semester, it should not have been unexpected. I hope for the future that these aspects of program will be more important to the members.
The among FPLC assessments of the program components are only important only in so far as they can help us improve the program for the future. In particular, the seminar/meeting result may help us improve this component if we can determine the aspects that contributed to their success. Similarly, the nearly significant difference in rating of the CTLpedia may suggest some ways to improve the documentation of our program or at least that program elements should not be changed during the year.
Comments about the meetings…
Comments about CTLpedia…
FPLC Impacts on Professional Development
The highest impacts on professional development were the perspectives of teaching and learning beyond the perspective of your discipline. This again suggests the value of the connections formed within the TPLCs and in particular the value of the interdisciplinary interactions that occur.
The next highest rating was for the view of teaching as an intellectual pursuit. This corresponds to our focus on scholarly teaching and deep consideration of the literature related to each FPLC topic. Also relatively high was the rating of research and scholarly interest in your discipline, which indicates that FPLCs can further disciplinary considerations too.
The next two impacts with relatively high rating were understanding of the role of a faculty member at our college and comfort as a member of the college community. These impacts further suggest the value of interdisciplinary connections and learning from our colleagues. Moreover, both of these exhibited significant increases in the paired comparisons tests indicating the impact of the program on these development areas.
Significant pre/post gains in professional development were seen for understanding of the role of a faculty member, comfort as a member of the college community, and teaching and learning as an intellectual pursuit. The first two support the impact of our program on connecting faculty and staff to the college community, whereas the last suggests that we were successful in engaging the faculty and staff in the study of teaching and learning.
Table of pre/post t-tests…
The most significant among FPLC difference using the pre-participation score as a covariate was the comfort as a member of the college community. In addition, nearly significant differences in the pre/post difference scores were seen in understanding of the role of a faculty member at our college (p<0.08) and ways of integrating teaching and research experience (p<0.08).
Need integrating teaching/res Diff score graph
When all post participation scores were included without the pre covariate, teaching as an intellection pursuit differed significantly among FPLCs (p<0.5), and total effectiveness as a teacher was nearly significant (p<0.09).
Comments about professional development outcomes…
Professional Development Discussion…
FPLC Impacts on Student Learning
Faculty perceptions of the ways student learning might change as a result of their FPLC participation were remarkably consistent across most of the 31 ways that students might learn more (at least those that were relevant to our FPLCs). Most ratings were between the students learn more to a small extent and more to a medium extent. None of the ratings increased significantly between pre and post assessments.
When a similar question asked about impacts on student learning from changes in the faculty member associated with their FPLC experience, the highest rating was given to the effect of their teaching project on student learning. In addition, this question was the only one that significantly increased in the post assessment over the pre- assessment. Mean responses for the other ways that faculty members might impact student learning gave relatively flat responses.
Conclusions
Connections goal
Colleagueship highest rated Facilitators highly rated and significantly increased from pre to post-assessment Highest impacts on professional development were the perspectives of teaching and learning beyond the perspective of your discipline relatively high rating were understanding of the role of a faculty member at our college and comfort as a member of the college community. both of these exhibited significant increases in the paired comparisons tests
Engagement in improving teaching and learning goal
Although the rating of the impat of the teaching project on the members decreased significantly pre/post, the rating ot the impat of the project on student learning increased significantly pre/post. next highest rating is for the view of teaching as an intellectual pursuit. relatively high was the rating of research and scholarly interest in your discipline
Conclusions
Categories: CTL Projects | FPLC | 2006-7 | FPLC Program
