Evaluations have gained in popularity in Canada since the 1990s, but statistical data indicate that the resources allocated to this management tool have not increased accordingly, despite the increased demand. During the same period, regardless of signify cant efforts to optimize governance, the Canadian federal governmentís management of issues related to Aboriginal peoples presents some weaknesses. Because evaluation may directly affect the administration of public programs, this study proposes a meta-evaluation of First Nations program evaluations. To do so, we replicate a methodology previously used by the Treasury Board Secretariat in 2004 to complete a vast study assessing the quality of evaluation in Canada. This article, based on the systematic analysis of a nonprobability sampling of more than 20 program evaluation reports, has applied the TBSís meta-evaluation techniques to the Aboriginal context. The results show that the evaluation of Aboriginal programs is of good, and even excellent, quality and suggest that the TBSís evaluation policy has had a definitive impact on evaluation quality.
Against the backdrop of severe and persistent social and economic disadvantage facing Indigenous Australians, this report reviews evaluations of the major labour market assistance measures for Indigenous Australians since the pivotal Miller Report of 1985. It highlights what are seen as failings in the evaluation of Indigenous programs over this time. History European settlement and subsequent capitalist economic development in Australia resulted in widespread destruction of the traditional economic and cultural activities of Indigenous Australians. Yet, as recently as the 1960s and 1970s, significant institutional barriers inhibited Indigenous integration with the mainstream economy. To the extent that Indigenous Australians do aspire to integration with the mainstream economy, they face the disadvantages inherent in being in the early phase of a profound cultural and economic transition while, at the same time, the ësafety netí of their customary way of life and their attachment to it are steadily vanishing. A major review of Indigenous employment and training programs delivered in 1985 (the Miller Report) challenged the assumption underlying early programsóthat integration with the mainstream or market economy was the best strategy for Indigenous people. Programs that followed contained an uneasy mismatch between the objective of respecting Indigenous choice and self-determination on the one hand, and pursuing equality as measured by mainstream indicators of labour market achievement on the other. The most enduring program embodying the concept of self-determination is the Community Development Employment Projects scheme, first established in 1977. The second major Indigenous-specific program has been the Training for Aboriginals Program. A range of other programs have been implemented under the ëumbrellaí policies of the Aboriginal Employment Development Policy and the Indigenous Employment Policy. Community Development Employment Projects scheme The story of the Community Development Employment Projects scheme provides ample demonstration of the failure of the evaluation effort to genuinely support the notion of self-determination and to value the preservation of Indigenous culture. The schemeís objectives, as originally stated and restated through the 1980s and 1990s, were to reduce the adverse effects of unemployment and welfare dependency, to strengthen communities, and to promote self-determination and cultural maintenance. In recent years government policy has increasingly refocused the objectives of the scheme onto unsubsidized employment outcomes. A central tenet of any program evaluation methodology must be to link objectives, implemented processes and measured outcomes. However, evaluations of the Community Development Employment Projects scheme over the years have focused upon paid employment outcomes as the measure of success. The objectives of self-determination, community capacity-building and cultural maintenance have never received appropriate support through the normal processes of policy development and refinement, informed through evaluation. Some studies have identified a number of positive effects of the scheme for communities, including improved social and cultural cohesion, NCVER 9 reduced incidences of alcoholism and incarceration, and greater capacity for self-management; others argue that the scheme represents a poverty trap. Aboriginal Employment Development Policy A similar story can be told about the Aboriginal Employment Development Policy, introduced in 1987 in response to the Miller Report. The language of the policy implied that promotion of self-determination and cultural preservation were key objectives. Accordingly, Indigenous people themselves were to exercise significant influence in formulating the objectives of labour market programs to ensure their alignment with Indigenous values and aspirations. Despite this, the formal statement of objectives consisted of a series of targets more consistent with those of assimilation. Again, the broader strategy was undermined or neglected through a failure to offer any outcome measures aligned with its stated principal objectives of self-determination and cultural maintenance. Current policy environment The current Australian Governmentís political agenda now openly pursues the integration of Indigenous people and communities into the market economy, and indeed this is a legitimate and important objective for some Indigenous people. The Indigenous Employment Policy emphasizes employment, and mainstream employment in particular, as the primary objective, with little discussion of the limited applicability this must have for Indigenous people in remote communities or those who wish to pursue traditional lifestyles. A number of Indigenous-specific programs have been surprisingly effective in boosting employment. The Training for Aboriginals Program, along with the main Indigenous-specific labour market programs that replaced it, appears to have been very successful in promoting employment opportunities when considered in the context of the effectiveness of labour market programs more generally. Patchy as it is, evidence suggests that a mix of on-the-job work experience, achieved through wage subsidies or brokered placements, combined with other appropriate support, such as mentoring and training, offers the most successful approach to achieving market employment outcomes for Indigenous job seekers. In terms of participation in mainstream labour market programs, Indigenous clients were well represented in referrals for assistance to labour market programs by the Commonwealth Employment Service prior to Working Nation, a policy introduced in 1994 to tackle long-term employment. However, little information is available on outcomes from these programs. The approach to evaluation improved markedly with the implementation of Working Nation, and experience with these programs continued to support wage subsidies as one of the more effective means of assisting Indigenous job seekers. While the competitive employment services market, the Job Network, initially failed to deliver adequate assistance for Indigenous Australians, measures to address this have been put in place in the most recent contract periods, including the introduction of more specialist providers. On available evidence, the Job Network appears to be as effective in assisting Indigenous clients as non-Indigenous clients, assuming that their intentions are to enter mainstream employment. Implications for evaluating labour market programs for Indigenous people Due to data limitations, our knowledge of what does and does not work in overcoming Indigenous disadvantage in the labour market is very limited. There is evidence of superior outcomes in a range of contexts when Indigenous personnel are involved in program or service delivery, but this is not a necessary condition for success. To avoid repeating mistakes of the past, it is critical that future evaluations of programs differentiate between participantsí aspirations, particularly those relating to cultural attachment and geographic remoteness, when attempting to connect the sources of disadvantage to processes and outcomes. Evaluations of Indigenous outcomes in the areas of education and vocational education and training (VET) have made a far more concerted effort to account for the range of aspirations and to more rigorously assess outcomes against stated objectives than has been the case with evaluations of labour market programs.