EVALUATION OF THE AALBORG EXPERIMENT

    The activity of understanding is,
    essentially, the same as
    that of all problem-solving

    Karl Popper

One must expect that the consequences of this educational model is that the new graduates are less experienced in solving standard everyday problems as they will appear in future employment. On the other hand, they are expected to be much better qualified to undertake large and complicated tasks, to combine insight from different fields, to analyze new problems and to make themselves acquainted with new fields to which the problems of practice are related.

The aim is a broad insight into and understanding of the connections between different fields and skills, in order to function in a society, which is becomming increasingly more complicated. In principle it can thus be ensured that the graduates have obtained the skills and experience to enable them to solve also the unknown problems of the future.

But some scepticism may be displayed: will the graduates be able to face the demands of the profession? Will the new graduates reach the academic standards? We will try to answer these and other questions through the evaluation of our educational system.

INTRODUCTION TO THE EVALUATION

The novel problem-oriented project-organized studies at Aalborg University were recently recognized as a real innovation in higher education as important as the Open University concept in the United Kingdom.

This recognition was given by an international panel, which had been commissioned by the Danish Ministry of Education to evaluate the international competitiveness of the Danish electrical and electronic engineering educations. This evaluation included evaluation of the project-organized studies in Aalborg and a comparison with traditional university education (Arnbak, Boom, Langmaack and Nelson 1993).

In connection with this evaluation a questionnaire was sent to a selected number of employers within the electrical and electronic industries (N=43). The purpose was to assess the employers' satisfaction with the qualifications of the engineers. The response was not satisfactory, since only 37% completed the questionnaire (Kümmel 1993).

Another international evaluation process was carried out by an international panel, which had been commissionned to evaluate the Danish civil and construction engineering educations (Christophersen, Coupe, Lenschow and Townson 1994).

In connection with this evaluation a questionnaire was sent to all external examiners in Denmark within civil and construction engineering. The response was 55% (N=801). They were asked to assess the strengths and weaknesses of the students, the examinations and the relevance of the educations in relation to the labour market outside the university. (Evalueringscentret 1993).

Also a survey of the graduate engineers was conducted. A test sample of 353 civil and construction engineers, graduated in 1988 or 1991, was selected and interviewed over the phone. The response was 63% (PLS Consult 1993).

Besides this evaluation the university itself has initiated several evaluation processes. One of these was carried out by the Faculty of Technology and Science at Aalborg University in 1989.

This evaluation included a questionnaire sent to all students who had graduated from the faculty in 1986 with a degree in Engineering. The response was 74% (N = 339). This evaluation also included interviews arranged for a random sample of graduates and employers.

The purpose was to determine, on the basis of 3-4 years of experience as graduate engineer, what knowledge and experience from the studies were being used in practice, whether the project-organized education provided the necessary knowledge and experience for the professional practice, whether there were difficulties arising from the project-organized education in the employment and whether the graduates and employers were satisfied with the Aalborg engineering programmes (Jensen and Wagner 1990).

At the same time evaluation was also carried out by the University, including a questionnaire sent to all the students at the undergraduate level at 5th semester and at the graduate level at 9th semester. The response among the engineering students was 54% (N = 814).

The purpose was to determine the students' attitude to project-organized education and the contents of the studies, the desirable and assessed qualifications of the teachers and the expectations to the preparation for the profession. Many of the questions were comparable with questions to the graduate engineers (Rasmussen 1991).

The Faculty of Technology and Science in 1991 investigated the interaction between project work and practice by selecting 40% of the academic staff (mainly in mechanical and electrical engineering) and asking them to complete a questionnaire concerning the projects they had supervised in the years 1986-91. This selection included approx. 650 master's theses and 66 PhD projects. The purpose was to determine whether the projects were carried out in co-operation with industry, to assess the possibilities of implementation of the students' solutions in industry and to determine whether the students' solutions were actually implemented.

In 1993 an evaluation of freshmans year was carried out by the Faculty of Technology and Science. This included a questionnaire sent to all the freshmen (responce 70%, N=529), their supervisors (responce 50%, N=60 and external examiners (responce 77%, N=35).

THE PROJECT WORK

The international panel in electrical engineering found that the project-organized approach appears to attract students and they identified students who had chosen Aalborg University owing to its project-orientation.

In the questionnaires the engineering students as well as the professional engineers were asked why they had chosen this particular education.

Undergraduate Graduate Professional
The education and its subjects 41 52 55
Interesting profession 48 40 31
Regular job and high wages 5 3 5
Other 6 5 9
100 100 100

Figure 7. Reasons for choosing engineering education (%).

There were two reasons for the students to choose Aalborg University, the education and the expectations to the profession, while possibilities of a regular job and high wages were out of focus in the selection process.

Since the professional engineers made their choice in the early 80s and the undergraduates in the late 80s, it is also possible to see some changes from educational to professional focus during the 80s.

In the questionnaire to the freshmen 60% of the freshmen mentioned the project-organized education as the main motivation for their choice of Aalborg University.

The international panels found that the students were very enthusiastic about group work and considered the group pressure to be a positive experience, but they also identified the formation of groups as a difficult and painful process. The freshmen found the co-operation between the students as their main problem in the project work.

The freshmen's involvement with project work was not seen to be as effective as it may in one of the international evaluation reports, since the students did not have the technical knowledge or tools to benefit fully from the experience. On the other hand, it was found that this could be outweighed by the benefits of the early establishment of a group culture, which is the mainstay of the educational programme.

The same assessment emerged when we asked the students to assess the best semester in their curriculum. The assessment showed that the students did not appreciate the first semesters, when they were in lack of technical knowledge and scientific tools.

In some way these answers also reflect the students' appreciation of the possibility to decide the contents and organization of the project work themselves, since at the early stages the project work appears to be prescribed exercises, which are used repetitively and thus are not really problem-organized project work in the usual sense.

When we asked the students to assess the possibility to decide the contents of the project work as well as the organization of the process, the graduate students found better possibilities than the undergraduates.

Undergraduate Graduate
Excellent 27 45
Good 58 47
Limited 12 7
Bad

    3

1
100 100

Figure 9. The possibilities to decide the content and organization of the project work (%).

Despite these differences, both the undergraduate and the graduate students found good possibilities to decide the organization and contents of the project work.

Concerning the balance between individual work and group work we asked the graduate students to assess the actual balance and express their desirable balance.

The results show that the students want a balance with less group work and more individual work than that which they experience in the programme.

One of the main reasons for choosing Aalborg University was the project-organized programme, but did the contents of the programme come up to expectations?

Undergraduate Graduate
Excellent

    21

25
Good 67 68
Limited 12 7
Bad 0 0
100 100

Figure 11. Agreement between the students' expectations and the programme (%).

The result we see is that most of the students found that (to a reasonable extent) the programme comes up to their expectations regarding questions and subjects they wanted to deal with during their studies to and more than 20% found it very satisfying.

This was also one of the findings of the international panels, which found that the students as well as the faculty were supportive of the programme and the organisation.

THE BALANCE AND COHERENCE IN THE PROGRAMMES

We asked the students as well as the professional engineers to assess the balance between project work and courses in the curriculum where half of the time was sent on project work and half of the time on taught courses.

Let us first take a look at the project work.

Undergraduate Graduate Professional
Too much 11 12 15
Sufficient 73 74

    78

Too little 16 14 7
100 100 100

Figure 12. Assessment of the weight of the project work in the programme (%).

The students as well as the professional engineers found the weight of the project work in the curriculum very satisfactory. If we take a look at the taught courses, we will see the same picture.

Undergraduate Graduate Professional
Too much 18

    16

11
Sufficient 76 81 76
Too little 6 3 13
100

    100

100

Figure 13. Assessment of the weight of the taught courses in the programme (%).

We also asked the graduate students about the actual and desirable balance between project work and taught courses in the programme.

To conclude, we seem to have achieved the optimum balance between project work and taught courses in our project-organized programme.

The students' actual activities were evenly distributed between project work and taught courses. The distribution was:

  • project work 38%
  • taught courses 36%
  • individual studies 26%

The total work load in a week was 52 hours for each student.

Concerning the demands of the studies on students these were mainly considered sufficient by the students, but often too diffuse.

Undergraduate Graduate
Sufficient

    52

    49

Too easy 7 15
Too demanding 9

    2

Too diffuse 30 29
Too narrow 2 5
100 100

Figure 15. The students' assessment of the demands of the curriculum (%).

Anyway, this uncertainty diminished during the years at the university. At 1st semester 70% of the freshmen found the demands in the project work uncertain.

We also asked the students to assess the technical coherence in the programme, and here most of the students found that the technical coherence was somewhere between good and limited. The feeling of diffuseness and lack of technical coherence is one of the weaknesses of the project-organized education's constant demand for innovation and up-to-date knowledge.

Undergraduate Graduate
Excellent 7 4
Good 46 43
Limited 31 37
Bad 13 13
Don't know 3 3
100 100

Figure 16. The technical coherence in the programme (%).

The graduate students assessed the balance of the programmes between specialization and general polytechnic subjects as perfect, whereas they found that the attempt to make specialization in engineering by restrictions on the themes within which students can choose their problems and projects limited their possibilities. They wanted more optional curricula.

THE SUPERVISORS' QUALIFICATIONS

The faculty at the university has a traditional academic background, with an emphasis on research. This means that in many ways they are self-educated as supervisors for the project groups, even if they have had some training in project-organized education. The questionnaire showed that they have succeeded at least to a certain degree. Only few were judged to be bad, the main part average and one out of four was judged to be excellent.

The students and professional engineers also assessed the most important qualities the supervisors possessed and the qualities the students wished them to possess.

Undergraduate Graduate
Excellent 23 24
Average 65 61
Bad 12 15
100 100

Figure 18. The quality of the supervision in the project work (%).

This assessment was made by the use of Miller's list of ten teacher-qualities registered in an investigation of students' assessment in higher education (Miller 1988).

Undergraduate Graduate Professional Undergraduate Graduate Professional
Desirable qualities Possessed qualities
Willingness to advise 26 28 26 36 40 31
Engagement in the subject 18 20 22 14 18 17
Ability to give precise and clear explanations 17 14 17 6 4 -
Mastery of the subject 15 13 11 19 15 12
Ability to start discussions 8 12 - 8 7 13
Interest in the students 7 5 7 6 5 10
Ability to involve the students 3 4 - 3 2 -
Ability to judge justly 4 2 - 5 7 -
Thorough preparation and planning of the education 2 2 - 1 1 -
Ability to speak to an audience 0 0 - 2 1 -
100 100 100 100 100 100

Figure 19. The desirable and possessed qualities of the supervisors (%).

There was very good overlap between the profile the students and engineers wished the supervisors to possess, and the qualities the supervisors actually possessed. Quantified, the overlap was 62%. The only important difference in the profiles was that the students wanted more precise and clear explanations than they got from their supervisors. On the other hand, they were ready to accept a lower degree of willingness to advise on the part of the supervisor than the supervisors actually offered.

The questionnaire showed that the supervisors spent 20 confrontation hours in all with the project group every semester, with one supervision or instruction session every week on average.

EXAMINATION AND RECOGNITION

The examination system in Aalborg lays much emphasis on project evaluation with a written project report, poster, a presentation of the project and the project report, all in groups. The graduate students judged the examination system to be somewhere between good and sufficient, while the professionals judged it as good.

  Graduate Professional
Excellent 14 22
Good 33 41
Sufficient 33 27
Bad 17 9
Very bad 3 1
  100 100

Figure 20. Assessment of the project examination system (%).

An international panel found that viewed through traditional university eyes, the time allowed for the provision of the necessary scientific tools through traditional lectures did not seem sufficient for an education well founded on scientific principles. On the other hand, this panel concluded that the system did appear to work.

With the increasing global economy the contents of courses being offered as fulfilling the educational requirements for professional engineers, will come under scrutiny.

The low contents of taught courses in the project-organized programme will seem to dictate a careful review of whether professional engineering guidelines are being met, in order to have the curriculum accepted by the international institutions governing the professional registration of engineers.

Concerning the examination system at Aalborg University the panel found that the traditional university examination system, in which lecture courses are usually graded and project work assessed on a pass/fail basis, is inappropriate in project centered education. It will convey the message to the students that the project work is unimportant and thus undermine the whole project-organized concept.

Instead this panel recommended that the university establishes in place mechanisms for better differentiation between students on the basis of project assessment.

The panel recognized that this recommendation may also have a significant effect on the group dynamics but found it important to strengthen the Aalborg degree in the international market.

If not, they assessed that the university will have a cloud of suspicion hanging over it, that a weak student may hide in a group and that technical leaders will not be fully recognized.

This recommendation has been followed by more emphasis on the individual examination, but still on the basis of the project assessment and with group examinations. Consequently we have experienced that it is possible to have much better differentiation between the students without undermining the project-organized concept.

This is underlined by a later questionnaire to the external examiners in civil and construction engineering. They were asked to assess the evaluation and examination system. The result was a high degree of satisfaction with the system among the examiners. 86% of the examiners assessed the system to be satisfactory, of which 31% assessed it to be excellent. Only 14% found it less satisfactory and none found it unsatisfactory. A special problem is the diploma, which the employers find difficult to understand.

ACHIEVED KNOWLEDGE

If we compare the composition of the knowledge and experience which is achieved and used by the graduate students in their project-organized studies and by the graduate engineers in their professional practice, we will see convincing agreement.

The use of computing, foreign languages, practical knowledge, experimental knowledge, management, economics, knowledge about natural and work environment, resources, culture and society has the same weight at the university as in industry. The only difference is that more emphasis is placed on theoretical engineering and science at the university and more emphasis on economies in industry, but this is only natural in view of the different natures and aims of university and industry.

However the graduate students wanted more economics, management etc. at the expense of pure engineering in the programme.

The surprising result of the investigation was the low priority of the theoretical sciences among engineering graduates. Industrial leaders have always called for more theoretical science in engineering programmes, whereas the investigation showed that computing and foreign languages are much more important to industry.

The investigation exposed great differences between the composition of knowledge and experience used in industry, by a consulting engineer and by an employee in public administration.

In industry, the most important elements were computing, foreign languages and theoretical engineering, while knowledge about the environment, resources, culture and society was less important. Practical knowledge and theoretical science were also given a surprisingly low priority.

For consulting engineers the most important elements were computing, practical knowledge, theoretical engineering and economics, while knowledge about culture and society were of low interest.

In public administration the most important factors were knowledge about the environment and resources, but computing and theoretical engineering were also considered to be important, while foreign languages were not interesting.

An investigation to determine the source of professional knowledge revealed that (after 3-4 years of experience after graduation) approximately half of the engineering graduates considered project work to be the main source, while approximately one fifth were of the opinion that colleagues were the main source.

Only a small number found that systematic educational courses (undergraduate, postgraduate, etc.) were the main sources, although almost half found that the courses in the engineering programme were a partial source.

In reply to where they had learned to apply their knowledge in their professional activity, the respondents said that project work was more important than courses, whilst about one fifth were of the opinion that they had learned more from colleagues.

PREPARATION FOR THE PROFESSION

The concept of the project-organized studies is based on the assumption that during their project work the students also learn to deal with theoretical and practical problems.

This was recognized by the international panels, which found it very gratifying to see that our project centered programme took the view that experience of the practical world was essential in the education of engineers. The panel were convinced that the project-organized education in Aalborg prepared the young people beyond the provision of technical knowledge alone.

They also observed, that the students appeared confident, regularly integrated theory with practice and felt that they were prepared to tackle a wide range of problems in industry. One of the panels concluded that the project-organized education provided graduates that industry could easily adopt.

This is in accordance with the questionnaire, in which a major part of the students and professional engineers felt they were well prepared for the profession.

It is also interesting to see that the graduate engineers had experienced that they were better prepared for the profession than the students felt they would be.

Similarly the graduate students felt they would be better prepared for the profession than the undergraduate students felt they were.

  Undergraduate Graduate Professional
Excellent 8 7 20
Good 49 52 65
Limited 27 23 13
Bad 2 5 2
Don't know 14 13 -
  100 100 100

Figure 25. Assessment of the preparation for the profession (%).

If we look at the different fields of specialization, there was general satisfaction with the suitability of the project-organized study as preparation for the engineering graduates for all the fields of the engineering profession. There were some few differences between the fields.

80% of the civil and structural engineers considered themselves to be well prepared, while almost 90% of the mechanical engineers were of the same opinion.

90% of the electrical engineers considered themselves to be well prepared but there was a difference between the electronics engineers of whom almost all were of this opinion and the energy engineers, of whom slightly more than 80% were of this opinion.

The very well prepared graduates were to be found mainly in mechanical and construction engineering.

The basis of these results is the project work in which the students mainly have to deal with theoretical problems from real life, outside the university environment, and interact with practice.

The investigation of the master's theses showed that 62% were prepared in co-operation with private companies or public bodies. 41% of the solutions in these theses were assessed to be suitable for implementation and 20% of the solutions were in fact implemented in industry. 8% of the master's projects which were continued in PhD-projects.

The external examiners from industry and other universities assessed the contents of the civil and construction engineering programmes to be as satisfactory in relation to the labour market outside the university.

72% of the examiners assessed the relation of the programmes to be satisfactory, 17% of whom assessed it to be excellent. 28% assessed it to be less satisfactory, while none found it unsatisfactory.

Concerning the PhD-projects 66% were carried out in co-operation with industry or public works.

In the assessment of the PhD-projects 48% of the solutions were assessed to be suitable for implementation and 39% of the solutions were implemented in industry.

As expected, there was strong interaction with private companies and public bodies during the project work of the students and a considerable part of the students' solutions to the problems they work with in the project work was assessed to be suitable for implementation in industry, increasing with the students' progress in the curriculum.

On this background it is surprising that three out of four students as well as professional engineers assessed the amount of practical experience to be too limited in the programme.

If we compare the balance between theory and practice as well as between fundamental theoretical knowledge and textbook knowledge in the programme, the survey showed that the graduate students wanted more textbook knowledge and practice at the expense of theory and fundamental theoretical knowledge, due to the theoretical problems they deal with in their project work.

However in general the graduate engineers had not experienced difficulties in their first employment.

The graduates had no problems with their theoretical knowledge, neither did they have problems with economics or statutory conditions, or in working with other specialized fields than their own study specialization. Nor did they have problems with their colleagues.

This is emphasized by the fact that less than 1% had changed jobs in the course of the 3-4 years since their graduation, on the grounds of their feeling of professional insufficiency while 13% had changed jobs because of lack of challenge in their employment.

The only difficulties which were uncovered by the survey were that 30% of the civil and structural engineers had felt a lack of practical experience in their first employment, while 20-30% of the other engineers had felt difficulties with organization and the company culture in their first employment.

To conclude, the graduate engineers only felt minor difficulties in their first job and could easily be adopted in industry. This is underlined by the fact that 10% of the engineering graduates felt confident in their first employment after three weeks, rising to 40% after two months and 80% at the end of six months.

PROJECT-ORGANIZED VERSUS TRADITIONAL EDUCATION

The international panels assessed the work conducted by the students during their theses. They found it of a quality equivalent to that of institutions with which Aalborg University is competing internationally.

But the international panel also found that engineers graduating from Aalborg University had significantly different skills when compared with products from a more traditional education.

The project-organized education was found to be regarded as a complementary educational system serving slightly different needs both for the student and the market than those served the traditional education. The emphasis on synthesis and the group culture at Aalborg University generate a graduate more readily adaptable, and thus more directly employable. On the other hand, graduates of a more traditional system with an emphasis on analysis are perhaps better grounded in fundamentals and are more capable of working independently although in general they will require more on-the-job training. Aalborg University was found to put more emphasis on aspects of operational and interpersonal effectiveness compared with traditional institutions focused on fundamental concepts.

This was underlined by the questionnaire to the graduate engineers from Aalborg. They were asked to assess whether at some points the project-organized education had prepared them better or worse for their jobs compared with traditional engineering education.

Better Worse
Yes 59 36
No 25 51
Don't know 16 13
100 100

Figure 28. Assessment of engineers graduated in Aalborg of their preparation for their job compared with traditional engineering education (%).

At some points the graduates from Aalborg had experienced that they were better prepared, on other points worse. However, the number of engineers, who at some points had experienced they were better prepared for their jobs than traditional engineers, was twice as large as those who felt they were worse prepared. The areas in which more than 5% of the Aalborg engineers had experienced better preparation were:

  • Management and co-operation 29%
  • Project work and problem-solving23%
  • Communication skills 8%
  • General technical knowledge 5%

The areas in which more than 5% of the Aalborg engineers had experienced a worse preparation were:

  • Detailed textbook knowledge 10%
  • Fundamental knowledge 9%

To conclude, the graduates from Aalborg were stronger in management, co-operation, project work, problem-solving, communication and general technical knowledge, weaker in detailed textbook and fundamental technical knowledge.

A similar conclusion was drawn by the external examiners concerning thesis work and final examinations.

Figure 29. The final examiners' assessment of strengths and weaknesses of civil and construction engineering students from Aalborg University (AUC) and Technical University of Denmark/Denmark Engineering Academy (DTU). (N=114)

The final examiners from industry and other universities assessed the engineering students from Aalborg (AUC) to be stronger in problem solving, communication and general technical knowledge, while the traditional engineering students from Copenhagen (DTU) were assessed to be stronger in methodology and detailed textbook knowledge.

There were no differences between the external examiners' assessment of the level of the thesis work and final examinations at the two universities. The examiners found the scientific and professional levels at these examinations most satisfactory at both universities.

A slightly different conclusion was drawn by the employers of the engineers in their evaluation of the Danish electrical and electronics engineers. In a questionnaire (with an unsatisfactory response) the superiors of the employed engineers graduated from the two universities assessed there to be no differences between the general qualifications of the graduate engineers, while the graduates from Aalborg were assessed to have significantly better qualifications in co-operation.

But no other differences were assessed to be significant, including questions in analytic skills, technical knowledge and competence, in dependence and multidisciplinarity, creation of solutions and communication.

Finally no differences at all were found in a survey between the graduate civil and construction engineers.

This survey included both the assessment of the quality, level, content and topicality of the fundamental, scientific and technical knowledge in the programmes at the two universities.

Differences were only observed in the satisfaction with the educational methods and in the level of the courses tought.

More than 90% of the engineers graduated from Aalborg assessed the content of the project work in their education as sufficient and 80% assessed the contents of the lecture courses with a fixed curriculum as sufficient, while 50 % of the engineers graduated from the traditional engineering education assessed the contents of the project work in their education as too little and the contents of the lecture courses with a fixed curriculum as too extensive.

The level of the courses in Aalborg were in some cases assessed to be too demanding, in other cases not demanding enough. In the traditional education with courses with a more fixed curriculum, the demands were assessed to be more appropriate for the students.

This can be explained by the constantly changing lectures in project-organized education. A price we have to pay if we always want the latest and most relevant knowledge in the lectures.

The other Danish university which produces graduate engineers is Technical University of Denmark..

Both universities are state-financed, they have the same grants for each student, and the same requirements for their graduates.

This gives us the possibility to compare the effiency between project-organized and traditional engineering eudcation.

A comparison between the two universities shows that project-organized education generates higher effiency with a lower dropout ratio and the main part of the students graduate at the prescribed time. In fact 80% of our students in engineering pass their examinations.

    Parts of this chapter were presented in earlier versions at the Unesco Roundtable on Strategic Issues in Engineering Education (UNESCO/DIF) in Copenhagen 1992, the European Society for Engineering Educations seminar on Project-Organized Curricula in Engineering Education (SEFI/CDG) in Copenhagen 1993, the International Federation of Surveyors Commission 2 workshop on Educational Challenges (FIG) in Aalborg 1993, the International Conference on Teaching Science for Technology at tertiary Level (IVA) in Stockholm 1994 and the World Federation of Engineering Organizations 3rd World-Congress on Engineering Education (UNESCO/WFEO/ESE) in Cairo 1994.