EXECUTIVE SUMMARY
Two Council of Higher Education (CHE) studies (CHE, 2007) and (CHE, 2013) have explored the performance of students entering the South African higher education system in 2000 and 2006, respectively. The studies showed poor performance in terms of graduation and retention overall, but also that regardless of (i) institution, (ii) programme and (iii) programme category, black South Africans fared less well than their white peers. More recent cohort studies conducted by the Department of Higher Education & Training (DHET) appear to show some improvement in success, throughput and graduation rates, but the performance of the system is still far from optimal.
It is acknowledged that “The poor performance in Science, Technology, Engineering and Mathematics (STEM) Education and professional programmes is of particular concern, given the short supply of high-level skills in these fields” (CHE, 2013). Mathematics is fundamental to all STEM programmes.
Universities South Africa’s (USAf’s) National Benchmark Tests (NBTs) assess (i) academic literacy, (ii) quantitative literacy and (iii) mathematical ability. In particular the NBTs provide insights into the ways curricula should be adjusted to meet students’ needs. The University Capacity Development Programme (UCDP) proposal focuses specifically on support for students’ mathematical needs in higher education and in STEM programmes in particular, since this is of national importance.
There appears to be a lack of alignment between students’ literacy practices and mathematical proficiency, and the academic practices of their chosen disciplines. This gap results in significant attrition, which continues to widen inequality, and impacts the economy and future development. About 40% of all registered students have dropped out of their studies by the end of regulation time, and only 27% obtain their degrees in the regulation time (CHE, 2013). Even in institutions where forms of curriculum responsiveness such as mathematics curriculum-integrated support are found to be necessary, this is generally not available. However, there remains a clear national imperative to address the problem of high failure and drop-out rates, in STEM programmes, specifically in first year Mathematics courses.
The two CHE studies argued that appropriate placement of students and coherent curriculum response were essential in enhancing the performance of the system. A proposal for coherent curriculum reform has not yet been accepted by the Minister. However, support for the improvement of teaching and learning continues to be available in the form of ear-marked grants, such University Development Grants for interventions such as the University Capacity Development Programme. In addition, Foundation Programme Grants intended to fund Extended Programmes with an integrated foundation phase are also available.
At a meeting in November 2015, the USAf NBT Steering Committee discussed the need for research on the NBTs. Over the years, CETAP has developed a large data set of students’ scores on the NBTs. There is enormous potential to use this data for research which aims to (i) explore the use of performance in the NBTs to predict subsequent performance and (ii) to use this evidence of performance in the various test areas to inform curriculum design, particularly in the first year.
A paper presented at the Higher Education Summit convened by the Minister in October 2015 (DHET, 2015) argues for “the use of data to keep track of progress in these areas through a set of sophisticated, robust, reliable high-level indicators that are put in place for the sector and which also serve to measure progress in transforming the sector”. Tracking progress is critical, but the system also needs information which will facilitate improvement.
Where correlation analyses have been done, it has been within institutions that have been prepared to share their data. We are now proposing that DHET could make system wide data available for analyses.
We further propose (i) to use this data set to analyse how students’ performance in the NBTs correlate with graduation and drop-out rates across the system, and within individual institutions in different subject areas; as well as actual first year performance at individual institutions in different subject areas; and (ii) to research the effect of curriculum-integrated support initiatives for Mathematics in the first year. Briefly, this would involve the following:
- determine individual areas of need for students in identified institutions through sub-domain analysis of NBT Mathematics results in Year 1;
- in an at-risk course in identified institutions, determine what support for Mathematics has been available in the past (for example a hot seat, tutorial classes);
- based on what seems to have worked at other institutions, propose the inclusion of additional support (for example open educational resource material or other online topic-specific resources mediated by a blended learning specialist);
- at the end of Year 1 determine whether results are better than /same as /worse than before (even though this will be a new student group it should be indicative of the impact of the initiative).
The cycle would then be iterative, with new institutions considered in Year 2 and 3. This information should inform the nature of foundation courses and curriculum responsiveness, reduce university (especially first year) drop out, and increase throughput and graduation rates.
Student performance data across the sector is available from the DHET, which now has the capacity to conduct regular and reliable cohort studies (DHET, 2015). Institutional data would however also need to be sourced from individual universities. Efforts by researchers commissioned by the Steering Committee could largely be focused on data analysis rather than collection, given the availability of existing data sets. This analysis would require statistical expertise, as well as expertise in teaching and learning in higher education.
Test costs prevent some students from writing the NBTs, and prevent some institutions from requiring that they be written. The lack of NBTP Mathematics results for all higher education institutions is a limitation. In this proposal we ask that registered students, i.e. those already admitted, be funded to write the NBTs during the orientation phase. At a 2015 Universities South Africa meeting Prof Cheryl Foxcroft, an Executive Dean for Teaching and Learning at a large Comprehensive South African university, commented that “Anything less than a national effort would leave us with the limited capacity we presently possess and would leave us with the status quo of locally developed interventions based on minimal understanding of the diagnostic potential of the NBTs.” Reducing or eliminating the costs of writing the NBTs, especially with encouragement to institutions, from USAf, for the writing of tests, would result in a more substantial body of test information. Analysis of the NBT results would provide a clearer picture of the needs of the newly-registered students and a clearer idea of how their needs could be addressed.
The hope is that at-risk students (with risk identified by NBT results) are better supported through a targeted approach, informed by individual needs, that addresses subject knowledge and enhances the learning environment. At the end of the three year period this approach should show some improvement (measured by reduced drop out and increased pass rates).
MANAGEMENT STRUCTURE TO SUPPORT EFFECTIVE IMPLEMENTATION OF THE PLAN 2018-2020
The Director of CETAP reports to the Dean of CHED, who is accountable to the Deputy Vice Chancellor, who is accountable to Senate which is governed by the University Council.
The National Benchmark Tests Project has a Universities South Africa (USAf) steering committee which reports to the USAf Teaching and Learning Strategy Group.
MONITORING AND EVALUATION OF PROJECT IMPLEMENTATION AND RESULTS
The external evaluation of the implementation of the plan will be undertaken by the Universities South Africa (USAf) National Benchmark Tests Project Steering Committee with the aid of independent consultants.
PLAN OVERVIEW
Project Description and Scope
This University Capacity Development Programme (UCDP) proposal has two broad focus areas.
Firstly, analysis of NBT and Higher Education Management Information System ( HEMIS) data sets provide large amounts of diagnostic information data. This analysis will address how students’ performance in the NBTs correlates with graduation and drop out rates across the system and within individual institutions in different subject areas; also, within individual institutions how students’ performance in the NBTs correlates with their performance in the first year of their studies in different subject areas.. Secondly, an investigation into the type and extent of the available curriculum-integrated support for Mathematics , will identify best practices in this regard, and inform curriculum-integrated support initiative. In the current institutional climate, where campus disruptions and availability of resources are matters of concern, it will be important to consider how blended learning approaches can be implemented to optimise curriculum-integrated support.
The UCDP proposal would involve the collaboration of a number of different university types. Agreeing to participate in this project would initially imply that participating institutions would
- require (or at the very least encourage) their incoming students to write the NBTs either prior to or during orientation week
- engage individually with students identified as being at riskin a discussion regarding their NBTP (especially Mathematics) results
- share relevant information (as per collaborative agreement stipulations, which would be determined in advance, in consultation with the institutions) pertaining to curriculum-integrated support, with participating institutions;
- facilitate the arrangement of annual institutional teaching and learning workshops, at which NBT Mathematics sub-domain and diagnostic information could be presented, and suggestions for addressing shortfalls in curriculum-integrated support for Mathematics could be considered. Participation in this project would mean that the collaborative research into curriculum-integrated support for Mathematics, already undertaken by the five institutions participating in the Collaborative Mathematics Project (CMP 2016), could be used to develop a broader survey into support for Mathematics, with five institutions per year, for the following three years (i.e. 15 institutions would be targeted).
NBTP Mathematics data would be used at institutional, departmental, course and individual student level. At institutional level, a broad picture of the spread of Proficient, Intermediate and Basic scores would indicate the extent to which support should be provided. At departmental, course and individual level, analysis of diagnostic sub-domain test performance would indicate the type of specific curriculum-integrated support required (e.g. specific content issues, or the need for placement into a foundation programme).
Provision of meaningful information is achievable if, particularly in the participating institutions, all students write the NBT Mathematics test. Development of meaningful curriculum-integrated support is only possible if there is greater engagement with the test results at all levels, greater acceptance of the potential benefits of providing appropriate and targeted support, and greater collaboration on the types and extent of possible support.
Although all the NBTP tests have the potential to measure student competence in essential academic skills at the point of entry to higher education, the Mathematics test would appear to have greater national prominence, perhaps because of the immediate and measurable high failure rate in first year Mathematics courses, an apparent lack of alignment between school-leaving Mathematics performance (reflected in the results in the NSC subject Mathematics) and the need for improvement in enrolment in STEM programmes.
The analysis of the NBTP Mathematics test results has a major role to play in informing institutional response to students’ mathematical needs. These results have both a broad and more focused aspect. Broadly, the results provide benchmark levels (Proficient, Intermediate and Basic) which are indicative of the extent of the support institutions should be willing to provide (such as augmented programmes, or extended degree programmes for those students whose scores place them in the Intermediate or Basic categories); in a more focused way the sub-domain diagnostic analysis (analysis of the test results in terms of specific content areas that may need attention) can inform the type of support institutions need to provide (additional content where specific aspects of the curriculum have not been taught at school, or where significant lack of comprehension is evident; or other forms of support relating to learning styles). Some institutions have engaged meaningfully with the NBTP Mathematics results and there is significant potential for growing this aspect of the NBTP.
For many reasons (economic, political, institutional) not all prospective STEM students are currently required to write the NBTs. Of the 129 481 National Senior Certificate (NSC) Mathematics candidates in 2015 who achieved 30% or above, 53 039 (41%) wrote the NBT Mathematics test. Analysis of the NBT Mathematics results of these candidates found that 45% of those achieved levels six and seven on the NSC Mathematics examinations needed extensive support (i.e. these candidates’ scores fell into the Basic or Lower Intermediate NBTP Mathematics benchmark bands). This type of support is not generally available in higher education (Centre for Educational Testing for Access and Placement 2016). Furthermore, different universities respond differently to the available research informing potential student support. Lack of institutional engagement with NBT Mathematics results weakens the potential impact of the information.
The benefits of the diagnostic potential of the NBT Mathematics test would be significant if a larger number of institutions could
- make it a requirement that all their students planning to enroll in STEM programmes write the NBT Mathematics test;
- receive from the NBTP meaningful broad information (i.e. in terms of benchmark levels) and more detailed information (in terms of sub-domain diagnostic performance information) on an institutional, departmental, course and individual basis;
- share information on curriculum-integrated support initiatives emanating from the activities they have implemented in response to general and diagnostic information they have obtained.
How could the aim (outlined above) of the University Capacity Development Programme proposal be realised?
DHET making HEMIS data sets available to match to the NBT data sets would allow the analysis of the relationships between performance on the NBTs and subsequent performance such as graduation and drop out as well as performance in key first year subjects.
Reducing or eliminating the costs of writing the NBTs would result in a more substantial body of test information that would be available for analysis and could be used to inform the teaching and learning of Mathematics. Fully funding or subsidising the costs of writing the NBT Mathematics test would be highly desirable. Test costs may be preventing some students from writing the NBTs, and prevent some institutions from requiring that they be written. Institutions could then be provided with a more realistic picture of the needs of their prospective and actual students, and a clearer idea of how these needs could be addressed.
Expanding the current levels of curriculum-integrated support for Mathematics would be beneficial for institutions considering provision of appropriate support activities.
The analysis of large amounts of diagnostic test information, packaged for individual institutions, depends on a dedicated Mathematics test specialist and on significant statistical support. An experienced Mathematics researcher, test specialist and a statistician will be crucial.
There is a need to make the writing of the NBTP Mathematics test a requirement, which is not currently the situation. If there was no cost barrier to writing the tests, then the Universities South Africa view that NBT engagement is a national imperative, together with available CETAP research, should be sufficiently persuasive and effective.
Persuading institutions to engage with the results will require presentations late in 2017 and early in 2018, at university senior management level, by credible NBTP staff, supported by solid research and information. The presentations would include analysis of national NBTP MAT results (scores, benchmarks levels and sub-domain diagnostic information), and a focus on curriculum-integrated support currently available.
In a higher education climate where academics will of necessity have to cope with larger numbers of less well-prepared students, it is essential that the expertise of experienced academics is shared with their less experienced colleagues, so that resources relative to curriculum-integrated Mathematics support are not wasted in ‘re-inventing the wheel’. This is only possible if there is institutional collaboration across different institutional types. The project will seek to involve USAf Mathematics Community of Practice (COP), recent graduates and early career academic mathematicians to ensure that knowledge transfer and the development of teaching skills takes place. Such professional development and capacity building will benefit the higher education sector as a whole. It will also be important to consider optimal use of blended learning approaches to curriculum-integrated support.
Overall Objective/Results/Outcomes/Deliverables
Presentation of NBT Mathematics benchmark and sub-domain diagnostic performance information at institutional workshops will enable academics to engage with the needs of the students they have, rather than plan programmes tailored to students they expect or hope to have. Reliability of this data would be enhanced by a larger data pool. Investigating and then sharing current practices in terms of curriculum-integrated support for Mathematics would establish a baseline for such support, and facilitate further development and the identification of best practice in this regard. Some such support will of necessity need to be undertaken off campus, by students in their own time (with appropriate guidance). This could be partly achieved through the use of blended learning. It cannot be assumed that academics and students have the necessary skills to access and use blended learning effectively, and a blended learning specialist, who is also a competent mathematician, will be essential to this project.
1. The immediate objective would be to ensure a larger data pool of Mathematics information. The analysis of this, as well as an overview of current curriculum-integrated support initiatives, would be the focus of presentations to higher education senior management in late 2017 and early 2018.
2. The outcome of these presentations would be that institutions would be persuaded to use the information resulting from analysis of NBT Mathematics test results as a tool for placement and support, as well as the curriculum-integrated support initiatives identified, in determining the type and extent of the curriculum-integrated support that could be implemented.
3. The immediate institutional benefit should be that the initial (baseline) survey of curriculum-integrated support for Mathematics would provide a platform for engaging with other forms of support, such as blended learning. Greater information regarding curriculum-integrated support initiatives should guide institutions into making better decisions as to how best they might enhance the experience of their first-year students. Through a collaborative programme, academics, will be encouraged to make evidence-based decisions for curriculum change to enhance student success.
4. The primary long-term objective would be the benefit, to students, institutions and the higher education sector as a whole, from greater engagement with the mathematical profiles of the different groups of students in the different higher education institutions. They would also benefit from more extensive knowledge of curriculum-integrated support initiatives, including blended learning options. This should enhance the first-year experience, and reduce attrition.
[1]The project is supported through public funds, The Department of Performance Monitoring and Evaluation has published Guideline 2.2.3: Guideline for the planning of new implementation programmes, which should be used to plan the project, including its evaluation. Funds for the evaluation should be built into the budget for the project.
[2] A theory of change involves identifying a problem or a development need that a project/plan seeks to address, describing the existing situation with respect to the issue, describe what the desired situation is, and describing the steps that will be taken to get to the desired situation.