During 2010, 2011 and 2012,
approximately 20 colleges took part in developing and improving a nationally
normed standard test for the core elements of the IS Model Curriculum
published in 2010. This case study looks at one of those colleges and
presents how they have used their performance data for ongoing accreditation
purposes. The Information Systems Department at California State
University, Los Angeles (CSULA), submitted a standalone University Program
Review Self Study and an AACSB 5-Year Maintenance Report integrated into the
college’s AACSB report. Both groups commented on the quality of the
department’s reports and assessments primarily because of their five years
of IS Model Curriculum (ISA)
examination assessment data. This paper presents the accreditation and
assessment methodologies utilized and the complexity arising from the
different goals and objectives of the business college and the information
Key Words: assessment, accreditation, IS Model
Curriculum, ICCP, program review, normed testing
The following assessment requirements are quoted from
the three accreditation agencies which are impacting Information Systems
INSTITUTION LEVEL: Regional
“SLOs and expectations for student
learning at all levels; reflected in policies, advising, information
Systematic program review
includes SLOs, data on retention and graduation; includes reference to
external evidence and evaluators
Quality assurance processes; assessment and tracking; comparative
data; use of results to revise/improve”
PROGRAM LEVEL: National
1. What will
our students learn in our program? What are our expectations?
2. How will
they learn it?
3. How will
we know they have learned it or not?
4. What will
we do if they have not learned it? “
program must regularly use appropriate, documented processes for assessing
and evaluating the extent to which both the program educational
objectives and the student outcomes are being attained.
results of these evaluations must be systematically utilized as input for
the continuous improvement of the program.”
All Business Schools and the IS
Programs must meet or exceed relevant requirements if they hope to maintain
a quality accredited program and attract students in a competitive
The IS department at CSULA offers a BS
CIS degree and a MSIS degree that closely follows the IS Model Curriculum
from AIS and ACM. The Center for Computing Education Research (CCER)
offers nationally normed examinations that are direct measures of the IS
Model curriculum for 2002, 2010 and MSIS 2006 curriculums. The CSULA
department for IS has approximately 300 majors and twelve faculty members
and is a small department compared to the college’s 4,000 business students
and 70 business faculties.
The complexity of ACCSB accreditation for an IS
department in a business school rises from several factors including
differing AACSB and ABET IS
accreditation goals and objectives,
differing AACSB and ABET IS curriculum
and course structures.
differing AACSB, ABET IS, and
university program reviews and required outlines, and
differing college and IS department
outcome assessment goals and objectives.
changing faculty over assessment
periods and thus varied understanding of the needs and processes.
and difficulty of consistent reporting takes staff away from important
research and teaching assignments to focus on institutional reporting and
administration. A consistent model that reduces effort and provides for
changing faculty should be desired by all IS programs. This paper presents a
well tested and successful model.
The IS assessment complexity is
substantially reduced since the
CCER sponsored IS Model Curriculum exam results are reported by both IS
2010 courses and student learning outcomes and ABET IS learning outcome
objective structures. Research evidence by Snyder (2010) also supports the
use of the IS Model Curriculum exam. His research “compares ISA exam scores
earned by students in one particular Computer Information Systems program
with scores earned by the same students on the ETS Major Field Test (MFT)
and shows that the IS Model Curriculum Exam appears to measure knowledge
retention in the same manner as the MFT. Since the MFT is recommended as a
valid program assessment tool, the ISA Exam should be similarly
provides a detailed outline for the Maintenance Report (AACSB,
2012). Responding to the requirements in the standards and the
maintenance report outline generally involves two levels of committees.
College maintenance committee (a member from each
Department/program maintenance committees (2-3 members each)
Responsible for annual reports and 5th year maintenance
College outcomes assessment committee (a member from each
Department/program outcomes assessment committees (2-3 members each)
Responsible for annual reports and 5th year maintenance
College coordinator of accreditation activities (an almost full-time
data base software is essential to maintain the required accreditation
information and improve the efficiency of the committees. CSULA uses
Digital Measures software.
The college committees are
primarily responsible for the following sections of the outline,
1. Situational Analysis
2. Progress Update on
Concerns from Previous Review
3. Strategic Management -
College Policy Level Only
5. Assurance of Learning-
College Policy Level Only
6. Other Material
The department/program level
committees are primarily responsible for the following sections of the
3a. Department/Program level
mission, goals, objectives and strategic plan
5b. Department/Program level
outcomes assessment methods, measurements and feedback
Since the Information
Systems degrees or programs involve primarily items 3a and 5b, the scope of
this paper will discuss those topics.
2. IS DEPARTMENT MISSION
department’s mission is based on AACSB and WASC requirements and must be a
compatible subset of the university and college mission statements. The
CSULA statements follow.
L.A., a member of the California State University (CSU) system, offers
excellent and innovative educational opportunities to an urban student
population that reflects the diversity of the Los Angeles basin.”
“The College of Business and Economics
central mission is to be the educational gateway through which students of
diverse backgrounds emerge as graduates having the abilities and
sensitivities needed to be highly competitive in local, regional, and global
“The mission of the Department of
Information Systems is to educate and train a diverse population of students
for leadership roles in Information Technology (IT) by focusing on technical
competency, managerial and interpersonal skills, effective and ethical
decision making, and long-term professional growth in a dynamic field that
is globally oriented.”
requires that the entire self-study be derived from and compatible with the
mission statements. Extreme care must be used to assure this
compatibility. The department mission should be revised following each
five year regional accreditation cycle discussion, and approved by the
3. IS DEPARTMENT GOALS AND OBJECTIVES
Goals and objectives
implement the IS/IT mission statement. There are two sets of industry
standard goals and objectives available, from the IS Model Curriculums (AIS
& ACM) for 2002 and 2010 and from ABET the IS requirement for program
educational objectives (PEOs) and student outcomes (SOs) – student learning
outcomes and post-graduation employment outcomes.
ABET Student Outcomes
(called objectives by AACSB)
(a) An ability to apply knowledge of computing and mathematics appropriate
to the discipline
(b) An ability to analyze a problem, and identify and define the computing
requirements appropriate to its solution
(c) An ability to design, implement, and evaluate a computer-based system,
process, component, or program to meet desired needs
(d) An ability to function effectively on teams to accomplish a common goal
(e) An understanding of professional, ethical, legal, security and social
issues and responsibilities
(f) An ability to communicate effectively with a range of audiences
(g) An ability to analyze the local and global impact of computing on
individuals, organizations, and society
An understanding of processes that support the
delivery and management of information systems within business application
Objectives/Student outcomes are measured by the IS Model Curriculum
Examinations for 2002 and 2010. The CULA IS department has been giving this
test as a final examination to all of their IS majors for the last six
IS Model Curriculum Goals
IS professionals must have a broad
business and real world perspective.
- covers ABET objectives a,
b, and c
professionals must have strong analytical and critical thinking skills.
- covers ABET Objectives d,
e, and f
professionals must have interpersonal communication and team skills and have
strong ethical principles.
- covers ABET Objectives g,
professionals must design and implement information technology solutions
that enhance organizational performance.
- covers ABET Objective i
The following chart extracted from
CSULA’s 2011 Annual report illustrates the type of information available for
both Objective and Course assessment from the IS 2010 ISA
ability to design, implement, and evaluate a computer-based system, process,
component, or program to meet desired needs
= .9 (17 question points)
a = .7
Both national and CSULA exam means rounded to 45%.
4. IS DEPARTMENT STRATEGIC PLAN
expects refinements to the department strategic plan at least every five
years based on feedback from assessment and on external technical and
environment changes. Many strategic plans follow the maintenance report
outline. A typical outline using that approach follows. It is modeled on
several published AACSB related strategic Plans.
Typical Scope of IS Strategic Plan
Analysis (with SWAT Analysis)
Technology, Curriculum, Student, Faculty and Assessment related
Goals and Objectives
Processes and Outcomes
Financial Strategy Process
Feedback and Continuous
Strategic Initiatives and
related Action Plans
Curriculum, Student, Faculty and Outcomes Assessment related
CSULA’s strategic plan also
includes introductory material on Why Information Systems, and separate
sections on the BS and MS degree programs.
5. OUTCOMES ASSESSMENT
assessment/assurance (AOL) of learning as used in AACSB accreditation can be
“The systematic collection,
review, and use of information about educational programs undertaken for the
purpose of improving student learning and development.”
(Palomba and Banta, 1999).
Its scope is defined by AACSB in a white paper as
“The outcomes assessment
process should include
1. Definition of student
learning goals and objectives
2. Alignment of curricula with the adopted goals
3. Identification of instruments and measures to assess learning
4. Collection, analyzing, and dissemination of assessment information
Using assessment information for continuous improvement including
documentation that the assessment process is being carried out in a
systematic, ongoing basis.”
(AACSB Assurance of Learning
Standards: An Interpretation, 20007)
These AOL principals can be operationalized as
Accountability and Continuous Improvement.
This section illustrates how CSULA
implemented Accountability through use of a standard curriculum and the use
of the IS Model Curriculum Examination, and how it implemented Continuous
Improvement through feedback to all faculty annually of the examination’s
detailed course level results.
The remainder of this section is organized as shown in
the AACSB AOL Standard.
5.1 Demonstrate a high
degree of maturity in terms of delineation of clear learning goals
The College set up four
high level goals for each of the programs
Master Business Fundamentals
Master Analytical and Critical
Master Ethical, Interpersonal,
Communication, and Team Skills
Master a Functional Area
Therefore the IS
department was required to use them as is high level goals.
The college then produced
a standard set of subsidiary goals and objectives for tailoring by the
business-oriented functional areas. After an on-going discussion, the IS
department moved ahead by using the ABET standard goals and objectives
pattern set forth in Section 3. We argued that this pattern of goals and
objectives allowed us to do true qualitative and normed quantitative
assessment, the core of accreditation’s purpose.
5.2 Demonstrate the
implementation of outcome assessment processes
Through the use of their
committee structure and total faculty approval of their assessment actions,
they believed they were fully meeting AACSB requirements. The key is that
every faculty member annual receives a detailed tabulation of the
performance of their students by question area for each of their courses.
Sample ABET and IS 2010 course reports follows.
from an ABET Course Level Exam Report
Extract from an IS 2010 Course Level Exam
At the department level
CSULA have available the objectives level reports that give them a
quantitative evaluation of their overall program’s status. A sample follows.
from an ABET Goal/Objective Exam Report
reports are primarily for internal use. CSULA’s actual external assessment
document includes the following charts. They serve as the bases of their
evaluation of current annual status.
5.3 Demonstrate use of
assessment information to improve curricula
The following trend chart
shows the overall performance of the IS program since we began using of the
ISA Exit Exam.
included an overall explanation of why their scores were below average
“UCLA and USC skim the top 15% of Los Angeles basin high school students (a
California educational policy) leaving CSULA with the B and C students. “
therefore claim their performance is very good considering the pool of
available (remaining) students.
explains each of bends in the national and CSULA curves. For example the
national 2005-2006 drop was due to the addition of numerous programs
following the 2004 beta tests. The 2006-2007 gains represent the first set
of feedbacks. Also, it appears that the IS Model Curriculum Examination for
2010 is more difficult than the IS 2002 test. More faculty involvement in
review of test item level performance is required. Additionally, evaluation
and use by more colleges and universities broadens this nationally normed
(and even internationally normed) instrument increasing its value and thus
rise in 2009-2010 was interpreted as due to a curriculum change motivated by
their downward sloping scores – which could only have emerged from
consistent and continuous testing.
Closing the Loop
The key to maintaining quality is an emphasis on
feedback of quantitative performance to each faculty member. The preceding
chart indicates two significant teaching and curricular feedbacks that
improved CSULA scores. This type of feedback can be achieved only by the
continuous use of a national test such as the IS Model Curriculum
5.5 Continuous improvement
A major AACSB evaluation measure examines
continuous improvement. It is the opinion of the authors, that the use of
the feedback available from this universal IS Model Curriculum exam by
colleges and universities can be a major factor that will lead to their
success and positive assessment by accreditation agencies.
6. FINAL THOUGHTS
IS programs should consider themselves fortunate
by the continuing support of the standardized curriculums and certification
examination programs in our field. Few other fields have this advantage.
All of the IS oriented degree programs need to support these efforts by
periodically using the IS Model Curriculum Examination.
Approximately half of IS programs are outside
business schools and they may find the ABET criteria more consistent with
their engineering or science faculties’ approaches to accreditation. Since
the IS Model Curriculum examination provides direct ABET measures those ABET
accredited programs or those considering ABET accreditation should consider
using the IS Model Curriculum exam as their primary continuous assessment
and improvement tool.
AACSB Accreditation Coordinating Committee
(2007). AACSB Assurance of
Learning Standards: An Interpretation.
International (2012). Maintenance of Accreditation Handbook.
AACSB International (2012). Fifth-Year
Maintenance Report Outline.
ABET Computing Accreditation Commission (2012).
Criteria for Accrediting Computing
Programs, 2012 - 2013.
A. Palomba and Trudy W. Banta (1999).
Assessment Essentials: Planning, Implementing, and
Improving Assessment in Higher Education. Jossey-Bass Publishers, San
Information Systems Analyst (2012).
Institute for the Certification of Computing Professionals,
Des Plaines, IL 60018.