Development and Validation of an Instrument to Measure a Performance of Vocational High School

Some evaluation has been carried out in Vocational High Schools (VHSs), but most of it focuses on the documents or passive data rather than the processes happening in VHSs. Thus, this research focuses on the evaluation of VHS performances where the process is initiated with identifying the constructs and developing the instruments of performance evaluation for VHSs with Technology and Engineering expertise programs. Based on the problems found in the field, before conducting the evaluation, the researcher needs to, at first, develop the instrument of evaluation through three stages of development. The first stage is analyzing the concepts related to the evaluation by examining the factors affecting VHS performances. In the second stage, instrument development and instrument analysis (content validation) are conducted with the help of experts. The third stage is performance evaluation in VHSs located in Yogyakarta. The evaluation employs a set of instruments developed by the researchers. This research has produced a set of instruments for performance evaluation, which can be used extensively in VHSs. Based on the evaluation, it is found that the aspects of general management and academic management of the VHSs being studied are considered “Good”.


Introduction
In various situations and conditions, education is an important part of human life (Panina et al, 2019;Sirajudeen & Sulaiman, 2020;Suryavanshi, 2012). Generally, the cognitive aspect of human being will be better along with the high quality of education taken, although there has been some that are contradictory. In the broader scope, education plays a specific role in the life of a nation and country. The quality of a country is influenced by the education quality (Akareem & Hossain, 2016;Madani, 2019).
The Indonesian government has recognized the importance of education. From time to time, they try to improve the quality of education. It is represented in the issuance of various forms of regulations and legislation compiled by the executives and legislature. The legislation on education, which currently becomes the basis of implementing education programs is the Act No. 20 Year 2003 on National Education System (State Secretary of the Republic of Indonesia [SSRI], 2016b).
Vocational education currently becomes the focus of many people and institutions (Triyono & Moses, 2019). The government has enacted the policies to empower the vocational education by formulating the 2005-2025 long-term development plan which includes the Senior High School and Vocational High School students' ratio amendment, from 70:30 to 30:70. Moreover, there is an issuance of Presidential Instruction No. 9 Year 2016 on VHS Revitalization to improve the Indonesian human resource quality and competitiveness (SSRI, 2016a).
To be able to see the extent to which VHSs have carried out their duties and functions, evaluation is needed to be conducted. This evaluation is a necessity to see whether the current condition and actual performance of the schools, including the principals, teachers, educational staff, students, infrastructure, and facilities are in accordance with the regulations (Ramadhan et al., 2020;Retnawati et al., 2017). To obtain a good evaluation result, a set of good evaluation instruments is needed (Rossi et al., 2018).
An instrument is a tool that can measure an object in which the results are close to the real condition of the object. To obtain a good instrument, the factors influencing school performance need to be comprehensively analyzed (Ramadhan, Nasran et al., 2019;Ramadhan, Sumiharsono et al., 2019). Thus, the principles of a good instrument should be taken into account in the development process.
Based on surveys conducted in two qualified VHSs, school performance was evaluated through some aspects, namely the accreditation of the study program/expertise program, principal's work performance, and certified teacher's work performance. The study program evaluation is conducted by National Board of Accreditation for Schools and Madrasah by focusing on some aspects as referred to in the eight standards of education which are stipulated in the Government Regulation of the Republic of Indonesia No. 32 Year 2013 on National Education Standards.
The evaluation of the school principal's work performance is carried out by the Local Government Office of Education by considering seven main tasks and functions of school principals. Likewise, the teacher's work performance evaluation is conducted by the same institution and is called post-certification evaluation for teachers. Based on information from the principals, this evaluation is conducted only in some selected regions in Indonesia.
None of the above evaluation focuses on the activity done by schools and institutions. From the information obtained in the initial survey using the evaluation instrument, it is found that the most evaluation that has been conducted tend to focus on passive data. It means that they focus only on document analysis rather than process documentation. It needed an evaluation instrument which captures the efforts performed by schools in realizing their vision and missions, particularly VHSs with Technology and Engineering program. VHS is a vocational institution having many practicums activities. So, the purposes of this research are;  Identifying the constructs to design the performance evaluation instrument  To Develop and find out the validity of the content instrument to measure a performance of vocational high school which is developed.
 Conducting performance evaluation in two VHSs in Yogyakarta using the developed instrument.

Vocational Education
The term vocational education is often referred to as technical and vocational education. For example, in Kuwait, that kind of education is called technical and vocational education. The term is not much different from that of in Indonesia. Generally, VHS is education emphasizing the acquisition of knowledge, skills, and attitudes (Al-Ali, 2016; Beilmann & Espenberg, 2016;Billett, 2011;Lillis & Hogan, 1983).
Vocational education in one of the developed countries, America, started to develop in the early 1900s, amidst the debate on vocational training in public education. The general consensus raised the importance of vocational training as an alternative to American academic tradition. However, there was a different point of view on certain designs and implementation of public vocational education. Two figures, Charles Prosser and John Dewey, argued that the formed consensus was on the contrary to the real condition of vocational education (Rojewski, 2009).
Vocational education prepares people to work in the fields of commerce and arts. In addition, education generates professional assistants in the fields of engineering, accounting, nursing, medicine, architecture, and law. The field of art is usually based on user requests and is non-academic, but it is related to certain trades or commerce (Wikipedia, 2016).
According to several experts above, it is indicated that vocational education may take place in secondary schools, postsecondary school, diploma program, and higher education. It may also be integrated into internship programs. In the levels of post-secondary school, vocational education is often provided by an institute of technology, polytechnic, or a university (Wikipedia, 2016).

Vocational High School Performance
The focus of this study is the evaluation of vocational high school performance. However, since the scope of research is too broad, this study is limited so that the implementation is more directed and in-depth. Based on the explanation above, this study focuses on the effectiveness of schools in organizing resources, namely time, place, and human resources in providing students with good education. The effectiveness is shown by school performance, which is assessed both quantitatively and qualitatively based on the objectives intended by a school as an organization (DeNisi, & Smith, 2014;Fallon et al., 2015).
Vocational high schools comprise of vocational education at the level of secondary education. As explained above, it prepares students to work in various engineering fields at the secondary level. When associated with the Indonesian National Qualification Framework for VHS, three-year VHS is worth level 2 and the four-year VHS is worth level 3. Those VHSs are under Directorate of Vocational Development of the Ministry of Education and Culture (Mardapi, 2017;Ministry of Education and Culture, 2016). This study focuses on VHS performance evaluation and the effectiveness of VHSs in organizing resources. Thus, they can produce graduates who are ready to work in the business and industrial work.
If associated with a school or an institution, school performance evaluation is a set of measures for various activities and values applied in a school. The results of measurement are used as feedbacks that provide information on the individual achievement of society of academicians or groups within a school organization (Chouinard & Cousins, 2015;Coates, 2009;DeNisi, & Smith, 2014;Fallon et al., 2015). In line with the aim of conducting school performance evaluation, this study focuses on evaluating the school performance in preparing students to work in business and industrial fields.

Research Goal
This study aims to evaluate Technology and Engineering VHSs performances, which starts by developing instruments of evaluation. The initial step is to develop an evaluation instrument using a research and development approach. This was taken because of the approach aim to produce something that can be implemented in the field being studied (van den Akker et al., 2006).
This study adopted and modified two development models by van den Akker et al. (2006) and, Gable and Wolf (1993). There were four stages of developing the instrument; namely, conceptual analysis, construct identification, instrument development, and content validation. After the instrument was developed, it is used to evaluate the performance of two VHSs in Yogyakarta.

Sample and Data Collection
The product developed in this research was administered to some VHS components, namely students, teachers, and educational staff who provided information on school performance. To validate the content of the instruments, experts were invited to a Focus Group Discussion. The table below shows the number of the sample taken from each research participant. Student 303 Based on the information stated in the Ministry's Education Data Center, the researcher decided to choose public vocational high schools SMKN Yogyakarta and private vocational high schools SMKS Yogyakarta. Those schools were selected because they conducted Technology and Engineering expertise programs. Moreover, both schools had students meeting the requirements for analysis. At last, in terms of distance, those schools could be afforded by the researcher, so the process of collecting data from those schools could be done well.
The trials of research instruments found that the research participants from a SMKN Yogyakarta were 15 teachers, 10 educational staff, and 153 students majoring in Electric Power Engineering who were in grade XI and XII. Those students were selected because they had taken part in the Field Work Practice. Meanwhile, in SMKS Yogyakarta, 12 teachers, 5 educational staff, and 150 students from the Technology and Engineering expertise program of Electric Power Engineering, Computer Networking Engineering, and Light Vehicle Engineering were included in the research because they had taken part in the industrial internship.

Content of Validity
This validity is determined by using the experts' deal. The expert of the field of study is deal or domain which is measured to determine the level of validity of content. This case is caused by instrument measurement such as a test or questionnaire which is valid proofed if the expert believes that the instrument could measure the mastering of skill which is defined in domain measured (Ramadhan, Mardapi et al., 2019). This analysing validity used Aiken formula.
Aiken formulates the Aiken's formula V to count content-validity coefficient based on assessment result from the expert panel as much as n people toward an item from the terms of how far the item represents the measured contract (Ramadhan et al., 2020). The submitted formula by Aiken can be shown below: Where, V = validity index item S = score applied, each rater reduced low score in category used (s=r-lo,  r = rater score choice and lo = low score in score categorizing) N = number of rater C = number of criterion/rating

Analyzing of Data
In the conceptual analysis conducted in the first stage, the instruments of evaluation were assessed by teachers, educational staff, and students. The questionnaires contained questions with four alternative answers arranged based on the Likert scale. After identifying the construct and designing the instrument, the researcher conducted a Focus Group Discussion. In this activity, the research team invited experts on educational evaluation and vocational education in order to obtain a good instrument.
The techniques used to analyze the data obtained during research were descriptive statistics. This analysis was used to obtain the level of VHSs' performance. School performance is an accumulation of general and academic management scores. The categories of score distribution are presented below.

Findings
The focus of the research is evaluating the VHS performances, which is initiated with developing instruments for evaluating. Basically, there are three stages to design the instruments. The first stage is conceptual analysis, which aims to investigate the factors influencing VHS performance. The result of the analysis serves as the basis of developing the instrument blueprint. The second stage is validating the contents of instruments developed based on the blueprint. This stage is conducted with the support of experts as the validators. The third stage is evaluating VHS performance using the developed instruments.

Conceptual Analysis (Stage One)
Research Objective 1: Identifying the constructs to design the performance evaluation instrument In this stage, the researcher analyses the concepts related to the themes of the study. The concepts being analyzed are vocational education, school performance, school performance evaluation, effective education, education for sustainable development, capability approach, entrepreneurship education, and internships. The conceptual analysis is conducted to identify the factors influencing VHS performances. In addition, since the main theme of the research is the development of instruments, the researcher examines the concepts related to instrument development, for example, the instrument development method, measurement concept, as well as validity and reliability. Based on some considerations, the researcher adopts and modifies two development models by van den Akker et al. (2006) and, Gable and Wolf (1993).
Based on the review of several references, the researcher obtains information that school performance is influenced by some factors. Thus, this research focuses only on things related to conditions and management, which are then classified into two, general management and academic management. The aspects of both classifications are presented below (Aiken, 1980;Fleming & Kleinhenz, 2007), the achievements of general management include  (1) • Teacher's consistency in implementing teaching approaches (2) • The implementation of good vocational education (3) • Management of fieldwork (4) • Effective time allocation for study (5) • Expectations of student achievement (6) • Routine evaluation of student's progress (7) The construct is illustrated in the following figure The construct of the instrument contains a blueprint that has been developed, and the blueprint is then used to develop questionnaires. The summary of the questionnaire and the source of data are presented in the table below. Content analysis is a set of procedures performed by experts to review a construct. The experts review the blueprint of the instrument, its content, and sources of the data for the instrument. Based on the data obtained in this study, content analysis is performed using the Aiken's method to obtain content validity. The approach taken in this step is the Delphi method. The number of experts involved in this series of Delphi activities is the same as the number of experts at the FGD for construct validation (13 experts). The summary of the results of the content analysis or content validation is shown in Table 3.  1  5  3  5  3  5  3  5  5  4  5  5  5  3 41  0.79 Valid  2  5  3  3  5  3  5  3  5  4  3  4  4  3 37  0.71 Valid  3  5  5  3  4  5  5  3  3  3  3  4  5  5 Table 3 shows that all items in the developed instrument are valid. It means that they meet the requirements of content validity. Based on the Aiken's index criteria, the items are valid since they range from 0.67 to 0.87. The limits of significance based on the number of raters, according to Aiken (Aiken, 1980;Aiken, 1985) as follows: the significance value of p = 0.048 requires the V index is more than 0.67 and for p = 0.006 requires the V index is more than 0.75.
The next stage is preparing the instrument based on the validated blueprint. The developed instrument is in the form of questionnaires and its scoring guideline. There are three sets of questionnaires made according to the intended respondents, namely teachers, educational staff, and students.

VHS Performance Evaluation (Stage Three)
Research Objective 3: Conducting performance evaluation in two VHSs in Yogyakarta using the developed instrument.
After the instruments of performance evaluation are compiled and validated, the next step was to evaluate the Technology and Engineering study program at VHS. This evaluation activity was carried out in two schools, namely one state school in SMKN Yogyakarta and one private school in SMKS Yogyakarta. The results of the evaluation for those are as follows.

Data Obtained from SMKN Yogyakarta
Results of evaluation which focus on General Management aspect in SMKN Yogyakarta are presented in Table 4.  Table 4 shows that the general management of SMKN Yogyakarta is considered "Good". It is shown from the average score of 2,905. However, when examined further, there are some aspects that need to be improved in terms of quality, namely Partnership management and Services for students.
For the aspect of academic management in SMKN Yogyakarta are shown in Table 5. It can be seen from the table above that in the aspect of academic management, the school performance is considered "Good". It is shown from the obtained value of 2.984. However, when further examined, the fifth aspect, which is related to time allocation for study, needs to be improved.
Overall, the performance of SMKN Yogyakarta in the aspects of general and academic management is categorized Good as it is shown in Table 6.

Data Obtained from SMKS Yogyakarta
This section discusses the results of evaluations conducted at SMKS Yogyakarta. General Management Scores are shown in Table 7. Table 7 above shows that the general management in SMKS Yogyakarta is considered "Good". The average score obtained in the general management aspect is 2.920. However, when further examined, there are some aspects that need to be improved. They are partnership management, management of facilities and infrastructure, and recognition of the success of school members.
Evaluation results on the aspects of Academic Management at SMKS Yogyakarta are presented in Table 8. The table above shows that SMKS Yogyakarta performance is considered "Good". It is shown from the score obtained (2.864). However, there are several aspects that need attention, namely time allocation. There is a need to manage time efficiently for independent study when there is no class.
Overall the performance of SMKS Yogyakarta for general management and academic management aspects can be categorized as Good. The results of evaluation are presented in the following.

Discussion
The constructs obtained in this study are summarized in two aspects, namely aspects of general management achievements and academic management achievements. General management achievement aspects include eleven (11) indicators, namely: leadership, involvement of teachers and education personnel in decision making, recording/documenting, monitoring progress, stability of staff delegation, overall staff development, recognition of success for every school member, partnership management, services to students, social climate management, management of facilities and infrastructure. Meanwhile the achievement of academic management include seven (7) indicators: organizing and implementing curriculum, teacher consistency in teaching approaches, applying good vocational education, managing work practices, maximizing the use of learning time, expectations on student achievement, routine evaluation of student progress.
This research activity has produced VHS performance evaluation instruments as follows: (a) instrument of VHS performance evaluation with teacher respondent, education staff respondent, and student respondent, (b) scoring guidance of vocational performance evaluation. The specifications of the instruments were (i) the instrument with teacher respondents composed of 26 items, (ii) the instrument with educational staff respondents composed of 20 items, (iii) the instrument with student respondents composed of 31 items. All items in the instruments are categorized as valid with Aiken's V index between 0.67 and 0.87 which assessed by 13 reviewers.
Based on the results of the study, the performance of two vocational schools in Yogyakarta for general management achievements and academic management achievements is categorized as good. Achievements of general management in state VHS is 2.905 (on a scale of 4) while in private VHS is 2.920 (on a scale of 4) and categorized as good. For the achievements of the academic management of state vocational high school obtained a score of 2.984 (on a scale of 4) while those in private vocational school got a score of 2,864 (on a scale of 4) and categorized as good.
The results of this study are relevant to the research of Cervai et al. (2013) which reveals that the external quality of learning outcomes in vocational education by considering a broader approach is to the perception and expectations of stakeholders. (Cervai et al., 2013). The findings of this research are also in line with the findings of Saunders (2012) which reveal that aspects carried out in performance include the design, implementation, and assessment of professional development initiatives (Saunders, 2012).
Likewise, the research conducted by Jaedun (2016) which shows that the results of school performance assessment based on the student's data source are consistent with the assessment results of the service provider, both the principal and the teacher. It is also in line with findings indicating that learners are one of the valid data sources for use in school performance assessments. The findings in this research also corroborate the research conducted by Jelantik et al. (2014) aims to know and analyze the level of performance of the school in SMK Rekayasa Denpasar seen from aspects and indicators that support the realization of effective school education management in SMK Rekayasa Denpasar.
The results of this research also confirm findings made by Yusmina and Murniati (2014) shows that the factors that affect, the first supporting factors: the quality of human resources teachers and the school principal is good enough, adequate infrastructure facilities, high level of community confidence and interest, students and school committees, the inhibitory factor: among others human resources is not optimal and the work culture of teachers and employees is not in accordance with integrated quality management.

Conclusion
Based on the results of the research, it is necessary to do several things so that the instruments have good quality and provide valid information related to the performance of the vocational high schools. The thing that needs to be done is expanding the scope of trials to obtain various information about the conditions of the schools. Besides, there is a need to prepare the implementation of evaluation more carefully −especially in the selection of respondents from the student component− so the students can fill in the questionnaires correctly.
The limitation of this study is the validation of the instrument of VHS performance evaluation set by expert judgments method. This validation is done by asking the opinion of experts to assess the accuracy of the instrument based on the aspects measured. Nevertheless the researchers believe that the validation results can be used as a basis for stating that the instrument has met the validity requirements, so that the instrument can be used to measure VHS performance.
Besides, the data validity of the instrument is highly dependent on the objectivity and seriousness of the respondent in answering the questions, In addition, the selection of student respondents is limited to students who have carried out fieldwork practices. Because, the information required needs such data which only those students provides. If any student is chosen it is believed that the data will be invalid because students have not been able to answer questions related to fieldwork practices