Final Report - Office of Internal Audit and Evaluation

Report submitted to the Parks Canada Evaluation Committee: January 11, 2017
Approved by the Agency CEO: March 16, 2017

Her Majesty the Queen in Right of Canada, represented by the Chief Executive Officer of Parks Canada, 2017
Catalogue No.: R62-414/2017E-PDF
ISBN: 978-0-660-07294-4

Table of Contents

  1. Introduction
  2. Program ntrDescription
    1. 2.1 Expected Results and Targets
    2. 2.2 Activities and Outputs
    3. 2.3 Resources (Inputs)
      1. 2.3.1. Budget and Expenditures
      2. 2.3.2. Human Resources, Roles and Responsibilities
    4. 2.4 Stakeholders and Partners
    5. 2.5 GCCP Logic Model
  3. Evaluation Design
  4. Evaluation Findings
    1. 4.1 Relevance
      1. 4.1.1. Continued Need for GCCP Program
      2. 4.1.2. Alignment with Government Priorities
      3. 4.1.3. Alignment with Federal Roles and Responsibilities
    2. 4.2 Performance
      1. 4.2.1. Outputs
      2. 4.2.2. Outcomes
      3. 4.2.3. Roles and Responsibilities
      4. 4.2.4. Efficiency and Economy
  5. Conclusions and Recommendations
  6. Appendix A: PCA Strategic Outcome and Program Alignment Architecture
  7. Appendix B: Evaluation Matrix
  8. Appendix C: Key Documents Consulted
  9. Appendix D: Evaluation of the Grant To International Peace Garden
  10. Appendix E: Evaluation of Grant to Support the Trans Canada Trail Foundation’s Fundraising Campaign

Executive Summary

This report presents the results of an evaluation of three separate programs: (A) the Parks Canada’s General Class Contributions Program; (B) the Grant to the International Peace Garden; and (C) the Grant to support the Trans Canada Trail Foundation’s Fundraising Campaign. The evaluation of these programs is required to meet the Agency’s commitment under the Treasury Board (TB) Policy on Evaluation (2009) and in accordance with section 42.1 of the Financial Administration Act (1985) to evaluate all grant and contribution programs over a five-year period. The evaluations are also consistent with the new TB Policy on Results (2016).

Evaluation Issues

Consistent with the requirement of the TB Policy on Evaluation and associated directive (2009), the evaluations addressed:

Relevance:
To what extent is there a continued need for the program? To what extent is the program aligned with government and Agency priorities and with federal roles and responsibilities?
Performance (Effectiveness, Efficiency and Economy):
To what extent are expected activities and outputs being completed? To what extent is there progress towards expected outcomes? To what extent is the program efficient and economical?

Scope & Methodology

Combined, the three grant and contribution programs evaluated account for less than 2% of the Agency’s total annual expenditures for the period covered by this evaluation (2010-11 to 2014-15). Related evaluation work was calibrated to the low risk and materiality of these programs. The evaluation of the GCCP was further calibrated to focus on key program changes since the last evaluation conducted by the Agency in 2010.

Data from multiple lines of evidence were collected for the evaluation of the GCCP, including: document and literature review; database analysis (i.e., financial and administrative program data); interviews with key Agency program administrators; survey of Agency program users; in-depth review of a sample of files; and comparative analysis using similar government programs as benchmarks. Data collection for the evaluations of the two grants was limited to a document and literature review. Parks Canada Agency evaluation staff conducted the evaluations’ field work between October 2015 and April 2016.

A. Evaluation of the General Class Contributions Program

Parks Canada’s General Class Contributions Program (GCCP) provides a funding authority to Agency managers (referred to as “program users”) to assist recipients in delivering a broad range of activities and projects that will support the Agency in fulfilling its mandate. The GCCP is not a “program” in the traditional sense as it does not represent a specific activity or sub-activity in the Agency’s Program Alignment Architecture (PAA). Rather, it contributes to a broad scope of programs and sub-programs across the PAA. Over the period covered by this evaluation, expenditures for the program averaged $8.2M per year. The GCCP’s Terms and Conditions (2015) outline key elements of the program including: objectives; expected results and outcomes; eligible recipients, projects and expenses; funding criteria; and application requirements.

Findings

Relevance

Overall, we found that the GCCP is relevant and consistent with the priorities, roles and responsibilities of both the Parks Canada Agency and the Government of Canada. A continued need for the program is justified by the ongoing demand for the program and by the absence of other partnering/funding mechanisms within PCA that offer sufficient flexibility to meet the objectives of funded projects. Further, evidence suggests that GCCP funding enables projects that otherwise may not have taken place. However, the evaluation identified opportunities to increase the relevance of the program by broadening specific requirements outlined in the GCCP’s Terms and Conditions to better address the Agency’s ongoing and emerging corporate priorities.

Performance

Overall, we found that the prescribed process for entering into and carrying out a contribution agreement under the GCCP is being followed. Projects are being assessed for eligibility and approved, and relevant funding is being dispersed. These approved and funded projects are aligned with expected requirements detailed in GCCP’s Terms and Conditions, are generally implemented as intended, and produce expected outputs.

In 2005, the Agency established a Centre of Expertise for Grants, Contributions and Partnering to facilitate the contribution agreement process and ensure an oversight role in the administration of the program. This includes a responsibility for ensuring ongoing reporting on agreement-level results (i.e., outcomes and value for money). We found that at the agreement-level, program expectations for activity and performance monitoring and reporting have been established and are reflected in the final contribution agreements. While evidence indicates that program users are involved in monitoring projects, required reports (e.g., final activity and financial reports) are not being consistently produced by funding recipients and/or records of these reports are not consistently maintained by responsible program users. Further, final project evaluations which provide an assessment of results and outcomes of the agreements are inconsistently completed by program users.

The Centre of Expertise also plays a key role for ensuring a strategic and centralized functioning of the program (e.g., identifying areas for collaboration, minimizing duplication, and maintaining a centralized roll-up of results achieved with the GCCP). We found that this role is not consistently being fulfilled. At the program-level, record maintenance practices present significant limitations. Data integrity issues were found with both the GCCP’s administrative database and individual agreement records. These data issues impede the Centre of Expertise’s ability to effectively monitor compliance and track the progress achieved by GCCP agreements or to assess the GCCP’s performance at the program-level. The Centre of Expertise has yet to produce expected annual performance reports on the GCCP.

We also examined the extent to which the GCCP is efficient and economical. We found that for the period under review, overall GCCP expenditures were reasonably aligned with approved allocations. Evidence suggests that the majority of funding for contribution agreements has been dispersed as planned. While the administrative costs of the GCCP are not tracked, the estimated costs represent a reasonable proportion of the program’s total expenditures as compared to available benchmarks.

The program has made some efforts to improve the efficiency of the GCCP process by implementing a GCCP Annual Plan to obtain CEO pre-approval for upcoming projects within the year and by continuously updating, clarifying and simplifying requirements and templates for the GCCP. However, we found that the Agency is still lacking a standardized risk-based approach for assessing proposals to ensure that administrative requirements are aligned with the level of risk and materiality of individual projects. As a result, key components of the GCCP process (e.g., approval structure, establishment of monitoring and reporting requirements) continue to be cumbersome and ill-suited to lower value agreements.

Recommendations

Key issues identified through this evaluation largely pertain to the administration of the program and contribution agreements. As such, our recommendations focus on the program’s administrative processes and requirements.

In light of opportunities identified to increase the relevance of the GCCP and enable the program to better address the Agency’s ongoing and emerging corporate priorities by broadening specific requirements outlined in the GCCP’s Terms and Conditions, we recommend that:

Recommendation 1

The Agency’s Chief Financial Officer should review the GCCP’s Terms and Conditions to ensure that stated requirements, particularly as they relate to expected results and outcomes, are aligned with Parks Canada’s Departmental Results Framework (to be approved by November 2017). 

Management Response:

Agreed. The Chief Financial Officer Directorate will review the GCCP’s Terms and Conditions to ensure alignment with Parks Canada’s Departmental Results Framework. Should changes be deemed pertinent, revised Terms and Conditions will be submitted to Treasury Board for approval. Target: 2017-18.

The notion of a risk-based approach is an integral concept of the TB Policy on Transfer Payments (2012). Treasury Board encourages administrative and reporting requirements for applicants and recipients to be aligned with associated risks. This scaling of effort increases efficiency, enabling more attention and resources to be directed to areas of highest risk. We found that the Agency has yet to consistently assess project risks or scale its approach accordingly. Given that low materiality projects (under $50K) represent a high proportion of agreements funded under the GCCP, we recommend that:

Recommendation 2:

The Chief Financial Officer should implement a risk-based approach to the GCCP, putting measures in place to ensure that program requirements are aligned with the level of risk of the project being funded. This includes:

  • Developing and applying a standardized risk assessment tool for proposals, with clearly documented rating criteria; and
  • Reviewing the GCCP’s contribution agreement process to streamline processes (e.g., approvals, monitoring and reporting requirements) where risk is low.
Management Response:

Agreed. The Chief Financial Officer Directorate will implement a risk-based approach to the GCCP, including a standardized risk assessment tool and streamlined processes for low-risk projects. Target: 2017-18.

The Centre of Expertise’s capacity to ensure an oversight role for the program is significantly limited by various program data availability and data integrity issues at both the agreement-level and program-level. Given this, we recommend that:

Recommendation 3:

The Chief Financial Officer should review program-level performance monitoring and reporting needs and processes to ensure that: 1) they are aligned with the Centre of Expertise’s roles and responsibilities as they relate to ensuring the strategic and centralized functioning of the program; 2) they support a risk-based approach; and 3) that effective mechanisms to collect the required data are in place. As part of this review, the Chief Financial Officer should:

  • Consider the development and delivery of training courses/materials to ensure all programs users and recipients are made aware of GCCP requirements.
  • Develop mechanisms to ensure consistent recipient reporting (e.g., standardized templates).
  • Review the structure of current post-project evaluations and ensure that requirements to complete these evaluations are consistently met by program users.
  • Implement further quality assurance and quality control mechanisms to ensure that reliable program information is collected and monitored through the GCCP’s centralized administrative database and records. 
Management Response:

Agreed. The Chief Financial Officer Directorate will conduct a review of program-level performance monitoring and reporting needs and processes to identify necessary changes. Targets: Review completed, 2016-17; Changes implemented, 2017-18.

B. Evaluation of the Grant to the International Peace Garden

The International Peace Garden (IPG) Inc. is a non-profit organization established in 1930 whose purpose is to create and maintain a garden as a memorial to the peace between Canada and the United States. The dedication of the garden took place in 1932. As part of annual unconditional government grant initiated in the 1930s, Parks Canada provides $22,700 yearly to the IPG to aid with the cost of general operations and maintenance of the garden.

Our evaluation of the grant confirmed that funds are being dispersed as planned, required monitoring and reporting mechanisms are in place and intended outputs are being produced.

C. Evaluation of the Grant to Support the Trans Canada Trail Foundation’s Fundraising Campaign

Initiated in 1992, the Trans Canada Trail (i.e., The Great Trail) is a network of recreational trails that wind through every province and territory from coast to coast. In 2014, a matching fund to be administered by Parks Canada was created with the intent to support the completion of the Trail by 2017 (i.e., Canada’s 150th anniversary). Through this initiative, the federal government will provide one dollar for every two dollars raised by the Trans Canada Trail (TCT) Foundation. Parks Canada’s total contribution to this grant program is expected to be up to $25M over four years.

Our evaluation of the grant confirmed that funds are being dispersed as planned, required monitoring and reporting mechanisms are in place and intended outputs are being produced.

1. Introduction

Parks Canada’s mandate is to:

“Protect and present nationally significant examples of Canada's natural and cultural heritage, and foster public understanding, appreciation and enjoyment in ways that ensure the ecological and commemorative integrity of these places for present and future generations.”

The Agency is responsible for four major heritage systems:

  • 46 National Parks (NP) of Canada;
  • 171 National Historic Sites (NHS) of Canada (administered by the Agency);
  • 4 National Marine Conservation Areas (NMCA) of Canada; and
  • 1 National Urban Park (NUP) of Canada.

PCA carries out its mandate through five programs and nineteen sub-programs (see Appendix A for the Program Alignment Architecture [PAA]). This evaluation focuses on the General Class Contributions Program (GCCP). The GCCP is one of two transfer payment (contribution) programs administered by Parks Canada. [1] It is not a “program” in the traditional sense as it is not represented as a specific activity or sub-activity in the Agency’s PAA. Rather, it contributes to a broad scope of programs and sub-programs across the PAA. In total, the program accounts for about 1% of the Agency’s annual direct program spending.

The scope of the evaluation also includes the activities and results of two additional grants administered by the Agency: the Grant to the International Peace Garden and funding to support the Trans Canada Trail (TCT) Foundation’s Fundraising Campaign (see Appendix D and Appendix E).

The program was identified as a low priority in the 2015-16 Evaluation Plan. PCA conducted the evaluation as part of its commitment under the Treasury Board (TB) Policy on Evaluation (2009) and in accordance with section 42.1 of the Financial Administration Act (1985), which requires that all grant and contribution (G&C) programs be evaluated over a five-year period. The evaluation is also consistent with the new TB Policy on Results (2016). The last comprehensive evaluation was completed in 2010.

2. Program Description

The GCCP provides funding authority to Agency managers (from here on out referred to as “program users”) to address periodic programming needs that are best addressed by entering into contribution agreements with appropriate recipients. The GCCP’s Terms and Conditions (Ts&Cs; 2015) outline key elements of the program including: objectives; expected results and outcomes; eligible recipients, projects and expenses; funding criteria; and application requirements. Further information on these terms and conditions are presented in the sections below.

2.1 Expected Results and Targets

The intended objective of GCCP is to assist recipients in conducting activities and delivering projects that will support the Agency in fulfilling its mandate. Expected results/outcomes outlined in the GCCP’s Ts&Cs (2015) are as presented below:

  • Canadians recognize, appreciate and are engaged in the values of natural and cultural conservation;
  • Stakeholders are engaged in terms of interest and involvement of common objectives towards ecological or cultural integrity;
  • Parks Canada managers and stakeholders have access to a better knowledge base for informed decision making and dialogue on commercial, ecological or aboriginal issues of mutual interest;
  • Visitors are provided opportunities to purchase souvenirs or experience traditional meals (e.g. Fortress of Louisburg);
  • Heritage assets are protected, secured and researched; and
  • Targeted audiences are educated in such areas as ecology, safety and other issues.

These expected results and outcomes have remained relatively consistent over five years covered by the evaluation.

2.2 Activities and Outputs

The Agency’s activities related to the GCCP are limited to the administration of the program. The current process for entering into and managing contribution agreements under the GCCP is presented in Table 1. The same process is followed regardless of the size of the contribution.

GCCP Contribution Agreement Process

Step 1: Identification of Projects
  • The Agency does not advertise the GCCP externally. Potential projects are identified by: unsolicited proposals from outside the Agency; program users seeking to engage recipients; or flowing from other agreements.
  • Program users consult with the Chief Financial Officer (CFO) Directorate’s Centre of Expertise for Grants, Contributions and Partnering (see Section 4.2.1.1) to determine if the GCCP is an appropriate financial instrument.
Step 2: Assessment of Qualification under GCCP Terms and Conditions (i.e., eligibility)
  • Proposed contributions are assessed against the GCCP’s Ts&Cs (i.e., eligibility of recipients, alignment with expected results, recipient’s management capacity, value of other contributions, and work plan and associated costs). 
Step 3: Pre-Approval by Chief Executive Officer (CEO)
  • Contributions must be included in the GCCP Annual Plan approved by the CEO. If a contribution is not on the GCCP Annual Plan, the briefing note seeking CEO approval (Step 4) must be submitted via the associated Executive Director and Vice President.
Step 4: Preparation of Contribution Agreement
  • Using available templates, interested program users prepares draft contribution agreement and related materials (CEO letter, communications plan, etc.) for review by the Centre of Expertise, and then recommendation by relevant financial managers and functional leads.
  • Final approval is given by CEO (signed briefing note). Then the letter to the recipient (informing them of approval) can be signed by responsible Field Unit Superintendent (FUS) or Director.[2]
Step 5: Provision of Contributions
  • Approved contribution agreement is signed by program user with delegated authority and eligible recipient (copy provided to Centre of Expertise for centralized file).
  • Funds are issued as per terms of contribution agreement.
Step 6: Monitoring
  • Program users are expected to regularly monitor the activities and progress of contribution recipients. Monitoring can be operational (i.e., verification of physical progress) and/or financial (i.e., examination of costs and expenditures), and should include in-kind contributions.
  • Program users maintain a local file with a copy of reports, evidence of results, and details of project expenses and payments (no centralized reporting).
Step 7: Evaluation
  • At the end of the agreement, a post-project evaluation form must be completed by the responsible program user. A copy of the form is to be sent to the Centre of Expertise within 30 days of the end of the agreement.

Source: Contribution Agreement Approval Process under the GCCP (2015).

In the past, these activities differed slightly depending on the eligible recipient. A distinct process was in place for Cooperating Associations given their special relationship with the Agency. These registered non-governmental, non-profit corporations assist Parks Canada by providing services to the public and/or generating revenues to direct towards approved activities that support Parks Canada’s mandate. Compared to other recipients, these associations are unique in that they exist solely or largely for the purposes of supporting one or more of the Agency’s national parks or national historic sites. The Agency has since completed a review of this component of the GCCP and, as of 2012-13, has stopped the use of the distinct process. The last agreements under this process were initiated in 2011-12.

2.3 Resources (Inputs)

2.3.1. Budget and Expenditures

The GCCP is funded through general appropriations and special purpose funds, with the majority being from A-based appropriation. While the Agency’s Main Estimates indicate a notional allocation, there is in practice no set budget for GCCP. Rather, Agency managers convert a portion of their existing O&M to contribution funding in order to establish agreements under the GCCP authority. Data in the Agency’s financial system show that expenditures for the program averaged $8.2M per year over the five year period covered by the evaluation (from 2010-11 to 2014-15). Details of this analysis are presented in the section on program efficiency and economy (4.2.4).

2.3.2. Human Resources, Roles and Responsibilities

As referenced in Table 1, in 2005, the Agency established a Centre of Expertise for Grants, Contributions and Partnering to facilitate the contribution agreement process. The Centre develops and updates the GCCP’s Ts&Cs and ensures an oversight role in the administration of the program. The specific activities of the Centre include:

  • developing and implementing standardized tools, templates and checklists for use by program users;
  • being available to provide support and guidance;
  • reviewing and recommending contribution agreements (prior to their signature by the recipient) that have been developed and negotiated by the relevant program user, based on approval by the CEO;
  • overseeing the development by program users of the risk-based monitoring plans for each agreement;
  • monitoring and ensuring compliance with the Centre’s approved administrative and management standards and processes; and,
  • tracking results achieved by the agreements, in consultation with the relevant program users.

The Centre of Expertise is comprised of two employees: one manager (PM-06) and one business analyst (AS-06). Both are also responsible for/involved in partnering strategies. The Centre reports to the Director, Procurement, Contracts and Contributions and CFO. For the purpose of this evaluation, these four individuals are from here on out referred to as “programs administrators”.

In practice, authority to approve and amend contribution agreements (up to $1M) rests with the CEO.[3] Once the CEO has approved the project or initiative and eligible recipient, a responsible manager with delegated signing authority can sign the agreement and has authority to approve payments and specific types of administrative amendments. Subsequent responsibility for the management of individual contribution agreements rests with the responsible program users (i.e., directors, managers or delegated program officers). These program users each dedicate a varying proportion of their time to administering the GCCP agreements. 

A summary of roles and responsibilities by level of accountability.

Applicable Program User (Director, FUS, or VP) Footnote4
  • Receiving, assessing and recommending what proposals/applications to fund
  • Responsibility for developing and negotiating funding arrangement
    • (Section 32)
  • Planning and carrying out monitoring activities throughout the duration of the funding arrangement
  • Authority for approving disbursements related to funding arrangements
    • (Section 34)
  • Ensuring adequacy of monitoring activities
  • Monitoring, tracking, measuring and reporting on progress/results achieved.
  • Responsibility for ensuring value for money
Program Administrator (CFO, Director, Procurement, Contracting and Contributions & Centre of Excellence)
  • Ensuring that the negotiated funding agreement is consistent with TBS requirements
  • Planning and carrying out monitoring activities throughout the duration of the funding arrangement
  • Approving disbursements related to funding arrangements
    • (Section 33)
  • Ensuring adequacy of monitoring activities
  • Ensuring value for money
CEO
  • Authority for approving funding recommendations

Source: GCCP Results-Based Management and Accountability and Risk-Based Audit Framework Result (RMAF/RBAF; 2005).

2.4 Stakeholders and Partners

While the outcomes of contribution agreements can benefit a wide range of stakeholders, the direct reach of the program extends primarily to the program’s potential and actual contribution recipients. Eligible recipients, as described in the program’s Ts&Cs (2015), include:

  • Voluntary associations;
  • Management boards that represent local stakeholders or that were created under land claim or park establishment agreements;
  • Non-profit organizations (e.g. environmental groups, aboriginal communities);
  • For-profit businesses, in cases where there would be no profit or increased value directly derived from the transfer (new since 2015);
  • Provinces/Territories/Municipal or Regional governments;
  • Universities, colleges and other post-secondary institutions;
  • International organizations; and
  • Departments, agencies or ministries of foreign governments (new since 2015).

As indicated in the above list, two of these recipient groups were added to the GCCP’s Ts&Cs during the period under evaluation (2015). The impetus for these changes is further discussed in section 4.1.2.

2.5 GCCP Logic Model

The logic model provides a visual summary of the program description. A logic model showing the relationships between the GCCP’s inputs (i.e., human and financial resources), activities, outputs, reach, and outcomes was developed for the purposes of this evaluation. This logic model is shown in the following table. 

PCA Strategic Outcome:

Canadians have a strong sense of connection to their national parks, national historic sites and national marine conservation areas and these protected places are enjoyed in ways that leave them unimpaired for present and future generations.

  • Inputs
    • Financial Resources (approx. $8.2M per year)
    • Human Resources (FTEs)
  • PCA Activities

    Program Administration

    • Develop and update Terms & Conditions
    • Develop and communicate project approval/evaluation processes, tools and templates
    • Provide support and guidance
    • Oversee, monitor and report on overall program results
    • Perform data management

    GCCP Delivery

    • Identify projects
    • Review and approve funding requests
    • Provide contributions
    • Monitor project
    • Complete project evaluation
  • PCA Outputs

    Program Administration

    • Approved Terms & Conditions
    • Project approval and evaluation processes
    • Tools and templates (for processes)
    • GCCP Project Database
    • Program-level performance reports

    GCCP Delivery

    • GCCP Annual Plans (i.e., pre-approvals)
    • Project reviews and approvals
    • Contribution agreements
    • Products or services stemming from recipient activities[5]
    • Project monitoring and performance reports
    • Completed post-project evaluations
  • Reach

    Program Administration

    • PCA program users (Agency managers and delegated program officers)

    GCCP Delivery

    • Eligible recipients (i.e., voluntary associations; management boards; non-profit organizations; for-profit businesses; provinces, territories, municipal or regional governments; Universities, colleges and other post-secondary institutions; international organizations; Departments, agencies or ministries of foreign governments)
  • Expected Outcomes
    • Recipients are assisted in conducting activities and delivering projects that support the Parks Canada Agency in fulfilling its mandate.

3. Evaluation Design

3.1 Evaluation Purpose and Scope

The evaluation examined the relevance and performance (i.e., effectiveness, efficiency, and economy) of the GCCP, consistent with the requirements of the TB Policy on Evaluation and related directive (2009). This evaluation generally covers the period from 2010-11 to 2014-15. Parks Canada Agency evaluation staff conducted the evaluation’s field work between October 2015 and April 2016.

The scope of the evaluation captures all of the Agency’s relevant activities and results generated over the five-year period. The GCCP’s RMAF/RBAF (2005) outlines the specific evaluation questions to be examined and methodologies to be used by the program evaluation. However, considering the low risk and materiality of the program, the evaluation framework was calibrated to focus on key program changes since the last evaluation in 2010.

This report also includes an evaluation of two additional grants administered by the Agency, i.e., Grant to the International Peace Garden (IPG) and Grant to support the Trans Canada Trail (TCT) Foundation’s Fundraising Campaign. Related evaluation work was calibrated to the low risk and materiality of these grants. Further information and evaluation observations are provided in Appendix D and Appendix E.

3.2 Approach, Methodology and Limitations

The evaluation addressed 11 specific questions and 17 associated expectations related to issues of relevance and performance. The key questions are shown in Table 4. A more detailed matrix of evaluation questions, what we expected to observe, indicators and relevant data sources is found in Appendix B. 

Evaluation Issues and Questions
Relevance
  • To what extent is there a continued need for the program?
  • To what extent is the program aligned with government priorities?
  • To what extent is the program aligned with federal roles and responsibilities?
Performance
  • To what extent are expected activities/ outputs being delivered?
  • To what extent are expected outcomes being achieved?
  • To what extent are roles, responsibilities and accountabilities for program delivery clear and effective?
  • To what extent is the program efficient and economical?

3.2.1. Methods

The evaluation employed multiple methods of data collection:

The evaluation employed multiple methods of data collection:

Document Review:
  • A variety of documents were reviewed for the evaluation, including results of the past evaluation and management response, legislation, policies, corporate reports, and program-specific documents such as Ts&Cs, GCCP Annual Plans, process templates and reported results/ post-project evaluation forms. A list of documents is provided in Appendix C.
Database Analysis:
  • Analysis included a review of secondary data from Agency sources, including the Agency’s financial system (STAR) and the GCCP’s administrative database.
File Review:
  • Using the GCCP administrative database, we selected a sample of 42 completed agreements for an in-depth file review. This review was used to collect information on the GCCP process (e.g., project qualification and project monitoring) and project performance that is not available in the database. It also provided insights into the clarity of roles and responsibilities (e.g., extent to which the Agency’s process templates are effectively used).
  • This sample is not considered representative. The initial sample frame included 10 non-randomly selected “high value” files (i.e., $500,000 and over) and 18 randomly selected files. An additional 14 files showing suspected anomalies as a result of database analysis were also targeted for review.
  • Project users responsible for the 42 selected agreements were contacted via e-mail and asked to provide key information related to the agreement including project proposals, pre-approvals, agreements, financial information/invoices, monitoring and reporting documents (including post-project evaluation forms) as well as any other relevant documents in electronic or hard copy form. Up to two reminders were sent in cases where no response was received or specific documents were not provided. In the end, sufficient performance information to conduct an in-depth file review was provided for 35 of the 42 files.
  • In addition, a request was made to the Parks Canada’s centralized records facility to obtain the financial files for these agreements. Since 2014, all financial files are to be sent to this facility for centralized record keeping after a specific period of time following payments. A number of files were not located at this facility, nor by the project manager or their respective Chiefs of Finance and Administration. For these cases, information from the financial system (STAR) was used as a proxy to verify payments made.
Key Informant Interviews:
  • Key informant interviews (n = 6) with select Agency managers and staff responsible for the administration and oversight of the program were conducted to gather information across all the issues of the evaluation. These interviews focused on significant changes to the context or delivery of the GCCP since the previous evaluation, as well as ongoing challenges.
Survey:
  • An on-line survey of program users (i.e., Agency managers and delegated program officers) was conducted to obtain their perspectives on program relevance and performance (effectiveness, efficiency and economy). Contact information for potential respondents was obtained through the GCCP administrative database. Program users no longer employed by Parks Canada were not contacted. In some cases, the contact identified in the database referred us to more appropriate respondents; our survey sample substituted these alternate respondents for the original contact. The survey had a 36% completion rate (i.e., total of 44 responses). This represents an estimated 22% of all program users during the five-year period.
Comparative Analysis.
  • The evaluation examined comparative information and data on the program design and performance of relevant benchmarks (e.g., NHS Cost-Share Program, TB submission process, grant and contribution (G&C) programs of other government departments). This helped to benchmark best practices and standards in the activities and performance of related programs.

3.2.2. Strengths, Limitations and Mitigation Strategies

The evaluation’s survey experienced a completion rate of 36%. This response rate was deemed satisfactory considering the high turnover of relevant staff. The sample of respondents included a good distribution of newer and seasoned program users. As such, this gives us enough confidence that the views of program users are relatively well represented in our analysis.

Key informant interviews included all key program representatives responsible for leading the administration of the GCCP over the period under evaluation. While a number of these individuals were only recently appointed to their position at the time that key informant interviews were conducted, the inclusion of prior incumbents in key informant interviews provides assurance that respondents’ perspectives were collectively representative of both past and present program operations.

In light of the evaluation’s calibrated approach, we did not conduct recipient interviews. The evaluation relied on the results of the previous evaluation’s recipient survey (2010). This approach is considered valid as the program’s approach with recipients has not since undergone significant changes. We supplemented and validated this earlier data with views of program administrators and users as well as findings from our in-depth file review.

The Agency disagreed with the 2010 evaluation recommendation to review the administrative costs for the program; it estimated that the costs of gathering and reporting on the administrative costs of the GCCP would outweigh the benefits that could be derived from the information. Given this, the current evaluation could not obtain an accurate estimate of administrative costs for the program.

Data limitations were identified with the financial information contained in the program’s administrative database. At the time of the last evaluation, Finance had recently begun reconciling planned versus actual expenditures for GCCP on a monthly basis. However, in requesting the GCCP administrative database, the evaluation found that the financial information had not been updated for the past two fiscal years (2014-15 and 2015-16). While this financial information was updated by the program for the purposes of the evaluation, we found that the updated database still contained a number of erroneous financial entries and other types of data limitations (e.g., missing and inconsistently captured project data). It was beyond the scope of the evaluation to correct or compensate for these data issues. Rather, data analysis and reporting was done where feasible and supported by additional data sources. Specific limitations and data integrity issues are discussed where relevant within the report.

The sample for our in-depth file review was drawn from the full universe of completed contribution agreements. However, due to the fact that the program’s records are not centralized, the evaluation was not able to obtain all the required documentation for a number of cases in this sample (i.e., financial and/or performance information was not provided or received for 7 of 42 cases).[6] Where possible, this was mitigated by cross-referencing data with other existing sources (e.g., financial system entries) and seeking additional information.

Post-project evaluation forms assessing performance were only completed for a small number of contribution agreements. As such, the evaluation’s ability to report on project performance continues to be limited by a lack of centralized reporting. This gap was mitigated by obtaining additional performance information through the in-depth file review and survey.

4. Evaluation Findings

4.1 Relevance

4.1.1. Continued Need for GCCP Program

Expectation:
  • The GCCP addresses an ongoing need within the Agency.
  • Managers use the GCCP.
  • The GCCP does not substantively duplicate other G&C programs.
Indicators:
  • Trends in contribution agreements (i.e., number, value, etc.).
  • # of program managers using the GCCP.
  • GCCP contribution funding as % of total project value (to be extrapolated from file review).
  • Program administrators and users’ perception of need for GCCP.
The GCCP addresses an ongoing need within the Agency.

Parks Canada is responsible for conducting a wide range of program activities. The GCCP is designed to help the Agency achieve its objectives through recipients who have a vested interest in Parks Canada’s mandate and mission. Contributions are intended to support specific activities that extend the Agency’s limited resources by leveraging internal funds with those of others, and to support projects where PCA lacks expertise or where others share jurisdiction.

Program administrators and users agreed that the GCCP continues to address an essential need within the Agency. Specifically, they indicated that:

  • The extent of results achieved by GCCP recipients may not have been feasibly achieved by the Agency alone.
  • The GCCP provides a level of flexibility that cannot be achieved through any other available partnering/funding mechanism.
  • The GCCP yields additional benefits, including enhancing the Agency’s opportunities for engaging a partners (e.g. for Indigenous consultation and relationship building), increasing Agency visibility through public education and outreach, and internal capacity building.

Evidence supports the assertion that most projects could not have been implemented without GCCP funding. While the GCCP requires the amount of funding provided by the Agency to be the minimum necessary to fulfil the project objectives, the Agency does not track its contribution as a percentage of total project funding. However, the sample of contribution agreements (n = 35) reviewed showed that funding provided by the GCCP represented 50% or more of the total project value for over two-thirds (66%) of projects.[7] Approximately 20% of projects were funded entirely as a result of the GCCP. In the previous evaluation, recipients confirmed the importance of GCCP funding for their projects and indicated that “without the funding projects would not have been undertaken, would have been scaled back or delayed or the final project would have been of lesser quality”.

Similar benefits have been identified for comparable class contribution programs across federal departments (e.g., Environment and Climate Change Canada, Fisheries and Oceans Canada, and Natural Resources Canada). Related evaluations emphasized that these programs continue to be viewed as effective and efficient mechanisms aimed at achieving federal government results while minimizing administrative burdens. Specifically, because the financial allocation to most individual projects is small, it is considered more efficient to use general class contributions than create a new set of Ts&Cs for each project.

Parks Canada’s program managers use the GCCP.

There is clear evidence that the GCCP continues to support programming needs within the Agency. During the five year period covered by the evaluation (i.e., 2010-11 and 2014-15):

  • Approximately 200 Agency managers/program officers used the GCCP to fund 330 projects.[8] This is comparable to the previous five-year period (2006-07 to 2010-11) within which an estimated total of 344 new projects were funded.[9]
  • On average, the Agency entered into 58 new contribution agreements each year (ranging from 41 to 83); a significant proportion (43%) of projects consisted of multi-year funding agreements. In any given year, this means that the Agency is funding on average 98 new or ongoing projects.
  • The Agency disbursed approximately $39.4M of funding. As presented in Figure 1, over half (52%) of the agreements were valued at less than $50,000. About 5% of agreements were valued over $500,000.
  • In-kind contributions were provided for 51 projects, including: materials, goods, services, facilities or time (e.g., honoraria, food, vehicle, gas/mileage). Where indicated, the estimated value of these in-kind contributions amounted to $1.4M.[10]
Figure 1: Distribution of Contribution Agreements by Approved Value ($000), 2010-11 to 2014-15

 

 

Figure 1: Distribution of Contribution Agreements by Approved Value ($000), 2010-11 to 2014-15 - Text Version

< $25 (37 %)
$25 to $49 (15%)
$50 to $99 (18%)
$100 to $249 (18%)
$250 to $499 (7%)
$500 to $1,000 (4%)
> 1,000 (1%)

Source: GCCP Administrative Database (Dec. 2015).

The GCCP does not duplicate other Agency contribution programs or funding mechanisms.

The evaluation did not find any evidence that the GCCP substantially duplicates other Agency contribution programs or funding mechanisms. The GCCP varies from the Agency’s other contributions program (i.e., NHS Cost-Sharing Program) insofar as it is a funding authority available to all Agency managers and is not tied to a specific program area or initiative. The Agency’s NHS Cost-Sharing Program maintains its own set of Ts&Cs that explicitly outline distinct eligible recipients and project types. GCCP agreements offer the flexibility to address gaps in recipient eligibility and project type left by other funding arrangements.

4.1.2. Alignment with Government and Agency Priorities

Expectation:
  • The GCCP Terms and Conditions align with Government of Canada and Agency priorities.
Indicators:
  • Degree to which GCCP Terms and Conditions align with GOC Whole of Government Framework.
  • Degree to which GCCP Terms and Conditions align with PCA priorities.
Alignment with Government priorities.

The GCCP’s Ts&Cs (2015) are consistent with priorities in the federal government’s Whole of Government Framework (i.e., high-level outcome areas defined for the government as a whole). The following table shows a cross-walk between the expected results and outcome areas identified for the GCCP and relevant objectives for the Whole of Government.

Cross-walk of GCCP Expected Results and Outcomes with Whole of Government Framework
Canadians recognize, appreciate and are engaged in the values of natural and cultural conservation.
  • A vibrant Canadian culture and heritage.
  • A clean and healthy environment.
Stakeholders are engaged in terms of interest and involvement of common objectives towards ecological or cultural integrity.
  • A vibrant Canadian culture and heritage.
  • A clean and healthy environment. 
Parks Canada managers and stakeholders have access to a better knowledge base for informed decision making and dialogue on commercial, ecological or aboriginal issues of mutual interest.
  • A vibrant Canadian culture and heritage.
  • A clean and healthy environment.
  • Strong economic growth.
Visitors are provided opportunities to purchase souvenirs or experience traditional meals.
  • Strong economic growth.
Heritage assets are protected, secured and researched.
  • A vibrant Canadian culture and heritage. 
Targeted audiences are educated in such areas as ecology, safety and other issues.
  • A vibrant Canadian culture and heritage.
  • A clean and healthy environment.
  • A safe and secure Canada.
Alignment with Parks Canada priorities.

The GCCP’s Ts&Cs (2015) reflect the Agency’s priorities through set funding criteria that require eligible recipients to demonstrate that stated project objectives are consistent with relevant Parks Canada objectives and programming requirements. Consequently, in order to be approved, the contribution agreements must be aligned with one or more expected results and outcomes that support the Agency’s programming priorities.

Program administrators and users agreed that for the most part, the GCCP’s expected results and outcomes are appropriately aligned with the Agency’s priorities. However, opportunities for further alignment were suggested:

  • The GCCP’s outcomes were seen to be relatively narrow and not easily mapped against the Agency’s high-level corporate priorities at the time of the evaluation (i.e., Asset Investment; Conservation Gains; Canada 150[11]; and Connecting Canadians and Visitors to Heritage Places). For example, it was noted that the outcome for ‘souvenirs and traditional meals’ could be broadened to better support the Agency’s objectives for improvement of visitor experiences and revenue generation.
  • The GCCP’s expected results and outcomes could be realigned to more clearly target relationship building with Indigenous partners and stakeholders (e.g., supporting consultations, raising awareness and promoting efforts towards reconciliation). This suggestion would support the Agency’s response strategy related to the mitigation of the key corporate risk related to the management of ‘Indigenous Relationships’ and would also further align with the Government of Canada’s priority to achieve reconciliation objectives with Indigenous people.

It is clear that the program has recognized and made some efforts to alleviate this disconnect. For example, there is a data field in the GCCP’s administrative database to indicate how funded projects align with programs in Parks Canada’s Program Alignment Architecture (PAA). As shown in Figure 3, most GCCP projects supported one of three programs, i.e., Heritage Places Promotion and Public Support, Heritage Places Conservation and/or Visitor Experience.

Figure 3: Number and Value of Contribution Agreements for Projects Funded under the GCCP by PCA Program Areas (2010-11 to 2014-15)

 

Figure 3: Number and Value of Contribution Agreements for Projects Funded under the GCCP by PCA Program Areas (2010-11 to 2014-15) - Text Version
  Program 1: Heritage Places Establishment Program 2: Heritage Places Conservation Program 3: Heritage Places Promotion and Public Support Program 4: Visitor Experience Program 5: Heritage Canals, Highways and Townsites Management
Value of Contribution Agreements (%) 10% 30% 31% 39% <1%
Number of Contribution Agreements (%) 8% 32% 47% 28% <1%

Source: GCCP Administrative Database (Dec. 2015).[12]

Additionally, the program has recently modified their Ts&Cs to increase the program’s flexibility and alignment with Agency priorities. Two notable amendments to the Ts&Cs were made since the last evaluation (2010):

  • In 2014-15, amended to include for-profit organizations among eligible recipients.[13] This was intended to create further opportunities for PCA to collaborate with businesses in the environmental sector and tourism industry. Guidance on how to complete agreements with for-profits is still under development; as of August 2016, no contribution agreements have yet been signed with this recipient type.
  • In 2015-16, amended to expand eligible international recipients to include foreign governments. This enabled the Agency to integrate an existing practice under the GCCP authority.

4.1.3. Alignment with Federal Roles and Responsibilities

Expectation:
  • The GCCP is clearly aligned with PCA’s legislative and policy mandate.
Indicators:
  • Federal legislation, policies and directives indicate relevant roles and responsibilities.
  • PCA mandate, policies and directives indicate relevant roles and responsibilities.

Within the Agency, legal authority for the financial provisions of the GCCP stems from section 19(1) of the Parks Canada Agency Act (1998). As with all G&C programs, the GCCP also adheres to the applicable requirements of the TB Policy on Transfer Payments (2012) and the Financial Administration Act (1985).

As stated in the program’s Ts&Cs (2015), the objective of GCCP is “to assist recipients in conducting activities and delivering projects that will support Parks Canada Agency in fulfilling its mandate”. As such, it is directly aligned with the Agency’s legislative role in protecting and presenting “Canada’s national parks, national historic sites and related heritage areas (…) for this and future generations and in order to further the achievement of the national interest as it relates to those parks, sites and heritage areas and related programs”(Parks Canada Agency Act; 1998).

In addition, the Agency’s Guiding Principles and Operational Policy (1994) indicate that Parks Canada must

“work with a broad range of federal, provincial, territorial and municipal government agencies, the private sector, groups, individuals, and Aboriginal interests to achieve mutually compatible goals and objectives”.

This highlights the Agency’s leadership role in facilitating a shared responsibility to manage areas of national significance by cooperating with others to protect and present national heritage. As such, the GCCP becomes an appropriate funding mechanism that enables the fulfilment of this policy-driven role.

4.2 Performance

4.2.1. Outputs

4.2.1.1 GCCP Approval Process
Expectation:
  • Applications are being reviewed and approved.
  • Contribution agreements for relevant projects are signed with eligible recipients.
Indicators:
  • Evidence of project qualification.
  • CEO Pre-Approval (GCCP Annual Plan and briefing notes).
  • Evidence of other approvals (signatures).
  • Number of contribution agreements.

The steps in the process for entering into a contribution agreement under the GCCP are described in section 2.2 of this report (see Table 1). Our evaluation of activities and outputs related to each step is discussed below.

Step 1 and 2: Project Identification and Assessment of Eligibility

In the first step of the GCCP process, program users identify potential projects to be funded through the GCCP. These program users are then responsible for assessing the eligibility of their proposal against set criteria in the GCCP’s Ts&Cs. This is usually done in consultation with the Centre of Expertise which is ultimately responsible for overseeing the assessment. The five criteria required to be assessed for each proposed project include:

  • Recipient eligibility and capacity for project delivery (e.g., expertise, skill sets and prior experience);
  • Alignment of project objectives and GCCP expected results;
  • Management capacity of the applicant (e.g., strength of recipient’s governance structure, its financial management capacity and stability, etc.);
  • Proposed budget, i.e., that the total costs of the contribution are reasonable and the availability of other contributors (including the recipient), their level of commitment and their total contribution (financial and in-kind) are in line with policy requirements; and
  • Project design, i.e., that the work plan and associated costs are realistic and proportionate to the activities to be undertaken.

The GCCP’s administrative database provides some evidence of project qualification for all funded projects. This database includes fields to identify that the agreements: (1) were signed with eligible recipients; (2) were related to eligible projects; and (3) that projects aligned with objectives and intended results outlined in the Ts&Cs. However, as described in the program’s RMAF/RBAF (2005), the originating program user is responsible for ensuring that results of the eligibility assessment process are documented. These records are expected to be maintained locally and are not centrally collected by the Centre of Expertise. As a result, further information to support project eligibility assessments is not included in the program’s database.

Our in-depth review of 35 files found the results of the eligibility assessment process are not consistently recorded. Most program users noted that the eligibility assessment had been completed through a series of undocumented correspondences with the Centre of Expertise. Full evidence of a formal eligibility assessment was only provided for one project included in our file review. Partial evidence of eligibility assessment provided for an additional 13 files consists mostly of the recipient’s initial self-assessment of project feasibility and, in two cases, an Agency-generated list of “pros and cons” for the project proposal. 

One indicator that this screening process is working as intended would be the number of projects not approved following initial eligibility assessment (i.e., have any projects been rejected?). This analysis is not possible given available data. 

Step 3: Project Pre-Approval

Contributions require pre-authorization of the CEO before any commitments are made to the potential recipient. For this purpose, the Agency produces a yearly GCCP Annual Plan which lists all pre-approved GCCP projects for which a contribution agreement may be signed within the fiscal year. The process for the Annual Plan is usually initiated in January/February when the Centre of Expertise sends out a call letter to Agency managers for project proposals. Once obtained, these project proposals are reviewed by the Centre and submitted for CEO pre-approval prior to the start of the fiscal year.

The evaluation found evidence that a GCCP Annual Plan was developed and approved by the CEO for each relevant fiscal year (2010-11 to 2014-15). In total, 267 new projects received pre-approval through this process (average of 61 per year). Annual plans were also used to pre-approve 36 project amendments. This leaves at least 50 projects for which pre-approval was sought using alternate mechanisms. Evidence suggests that use of the GCCP Annual Plan as the primary mechanism for facilitating the pre-approval of GCCP agreements has increased over time.

Step 4 and 5: Final Approval and Signature of Contribution Agreement

Following pre-approval of projects by the CEO, responsible program users must complete a number of required documents to obtain final CEO Approval. These include: the contribution agreement; a briefing note to the CEO; a letter from Parks Canada to the recipient; the communication approach; the transmittal slip; and the information form for proactive disclosure. Many of these also require signature by a relevant senior manager before the contribution agreement can be signed with the recipient.

We noted that many agreements captured in the program’s administrative database lacked a record of one or more of these required signatures. It is not clear whether this indicates an absence of approval or just poor record keeping practices. Records of CEO approval were the most complete (available for 99%) of agreements.[14]

The majority (91%) of approved new projects listed in the administrative database from 2010-11 to 2014-15 resulted in a signed contribution agreement for which payments were made as of the end of 2014-15.[15] Table 6 presents the distribution of number and value of these agreements by recipient type. The majority were provided to non-profit organizations (58%) and Indigenous groups/communities (24%).

Table 6. Number and Approved Value of Contribution Agreements, by Recipient Type (2010-11 to 2014-15)
Recipient Type[16] Agreements Value ($000)
Number % Total Amount % Average Amount per Agreement
Non-Profit Organizations/Voluntary Associations 192 58% 25,699 58% 134
Indigenous Groups/Communities/Management Boards 80 24% 11,236 25% 140
Provinces, Territories and Municipalities 16 5% 968 2% 60
Universities, Colleges and Post-Secondary Institutions 30 9% 3,080 7% 103
International Organizations 12 4% 3,331 8% 278

Source: GCCP Administrative Database (Dec. 2015).

Table 7 presents the distribution of the number and value of agreements by project type. As project types are not explicitly identified in the GCCP administrative database, this categorization was completed for the purposes of the evaluation. Some projects may in fact be linked to more than one project type; however, for the purpose of this exercise, only the best fit category was selected.

Table 7. Number and Approved Value of Contribution Agreements, by Project Type (2010-11 to 2014-15)
Project Type[17] Agreements Value ($000)
Number % Total Amount % Average Amount per Agreement
Seed and interim funding to co-operative associations providing enhanced (value-added) services for visitors (services outside PCA’s direct delivery mandate). 27 9% 482 1% 18
Funding for Research on and in support of commemorative and ecological integrity issues. 50 17% 5,154 11% 103
Funding to promote education, outreach and tourism. 109 38% 20,243 45% 186
Operational support for national historic sites - for sites not owned or administered by the Agency. 5 2% 630  1% 126 
Support to Aboriginal groups to obtain expertise related to land negotiations, park/site management planning and relationship building. 40 14% 4,346  10% 109 
Support to Aboriginal groups for capacity development for economic opportunities and presentation of culture. 21 7% 4,645 10% 221
Operational support to management boards. 5 2% 1,014  2% 203
Community relationships/project specific support to municipalities within national parks to meet environmental or other standards set by the Agency (e.g. sewage, water treatment plants). 5 2% 321 1% 64
Funding for ecological protection to nature conservation groups re: land assembly and easements bordering national parks. 9 3% 1,873 4% 208
Operational support to not-for-profit organizations to help establish areas of national historic or ecological significance. 5 2% 1,413 3% 283 
Support to international organizations to support specific events or for knowledge and capacity building in fields of mutual interest. 12 4% 4,345 10% 362
N/A (could not be classified based on available information) 42 13% 861 2% 20,505

Source: Analysis of GCCP Administrative Database (Dec. 2015).

4.2.1.2 Analysis of Monitoring of GCCP Project Activities and Results
Expectation:
  • Project monitoring is taking place.
Indicators:
  • Evidence of project monitoring activities by program users.
  • Evidence of performance information collected (monitoring of results).
Program expectations for performance monitoring have been established.

Program users responsible for managing the contribution agreements are expected to ensure regular monitoring of the progress towards the completion of activities and achievement of results. The standards for monitoring are established by the Centre for Expertise and detailed in the GCCP’s RMAF/RBAF (2005).[18] These responsibilities include:

  • monitoring the progress of each project/initiative;
  • reviewing progress and other reports and the results of monitoring activities to provide a basis for approving/not approving the release of funds;
  • identifying any issues or concerns related to performance/progress and or financial expenditures and taking appropriate and timely corrective actions; and
  • identifying and collecting any overpayments that should be returned to the Crown.
Contribution agreements include provisions for recipient reporting but required reports are not consistently completed and/or available.

As part of their monitoring activities, all contributions agreements examined in our file review (n = 35) included provisions requiring recipient reporting (e.g., interim and final activity reports, interim and final financial reports, and/or interim and final deliverables). The extent of these requirements varies based on the nature of the project and level of funding received. Final activity and financial reporting were required by all agreements. Agreements for multi-year and some single-year projects required these reports to be produced at least once per year, if not biennially or quarterly.

Despite these provisions, a number of program users were unable to provide us with evidence that all required reports had been produced and submitted as required. To some extent, this appears to be the result of poor record keeping practices compounded by a high turnover of program users. Some program users were able to provide records but had to contact the recipients to re-obtain the original documents. In other cases, the recipient had simply not provided them with all of the required documents. In three cases, the program user could only provide an e-mail attestation (self-report) to confirm that products were delivered as planned (one with a supporting internet link to the final result).

In total, we were provided with both the final activity report and final financial report for 51% of reviewed files; 89% of files had either one or the other. Of the 18 large value projects (i.e., over $100K) included in our sample, five (28%) were missing the final activity report.

Where available, we found that these final project reports were inconsistently completed. The format of reporting varied significantly and did not always align with requirements in the RMAF/RBAF (2005). While all final activity reports presented at least a limited description of project objectives, activities and outputs, only 28% included a required discussion of barriers and challenges. Some form of evaluation of project reach and satisfaction (required for projects over $100K) was included for 70% of those projects for which a final report was provided.

Program users have implemented additional monitoring activities.

In response to the evaluation’s survey, all program users reported undertaking additional activities to monitor recipients’ progress on agreements. These were most commonly reported to include interactions via e-mail, phone and/or periodic meetings with recipients. Other, less frequently used monitoring mechanisms were reported to include on-site interactions and membership/attendance on management boards. By their nature, most of these activities are difficult to track. Related evidence (e.g., meeting minutes, e-mail records, etc.) was only provided for a small number of contribution agreements (i.e., 14 files).

The Agency lacks a standardized risk-based project assessment framework.

While it is clear that there are gaps in the Agency’s ability to demonstrate that all monitoring and reporting requirements are met under the GCCP, as previously noted, the majority of related contribution agreements are for projects with relatively low materiality (under $50K). The TB Policy on Transfer Payments (2012) stipulates that departments and agencies have a responsibility to ensure that the administrative requirements on G&C recipients (i.e., monitoring and reporting requirements) are proportionate to the level of program risk, the value of funding in relation to administrative costs, and the risk profile of the recipient. Given this, we evaluated the extent to which the Agency is applying an appropriate risk-based assessment to determine administrative requirements.

Consistent with findings of the previous evaluation (2010), we found that the Agency continues to support a risk-based approach. Specific monitoring activities and reporting requirements for each contribution agreement are determined based on a risk-based monitoring plan that is developed during the negotiation phase of agreements. Risk factors considered include:

  • the capacity of and prior Agency experience with the funded partner;
  • the extent of any third party delivery partners/funding recipients;
  • the complexity of the project;
  • the nature of planned activities (e.g., clear-cut projects versus more complex and/or “soft” projects);
  • the materiality of the agreement; and
  • the duration of the agreement.

Our file review found that risk-based assessments appear to be conducted for each agreement; each file contained at least some observations of estimated project-specific risks and project monitoring requirements that appear to be based on project cost and timeframe. However, the structure and rationale behind these assessments was not clear. The Centre of Expertise confirmed that there is no standardized risk-assessment framework/tool used to evaluate the specific risk of recipients/projects and determine a reasonable structure for corresponding administrative monitoring and reporting requirements.

In 2013, Treasury Board Secretariat completed an assessment of departmental implementation of reforms to grant and contribution programs.[19] This report found that most departments (78%) included in the assessment now tailor ongoing monitoring and oversight to ensure that recipient reporting requirements are proportionate to the risk rating of the recipient and the project. In many cases, this information is shared with the recipient, bringing greater transparency to the oversight process. For example, one federal department now uses a ‘Corporate Risk Management Framework’ for all their G&C programs to assess the level of risk, measured throughout the project life-cycle, to determine the extent of departmental monitoring and recipient reporting required.

Concerns were also noted by program administrators and users regarding some recipients’ capacity to provide a final activity report. It was suggested that some key partners (e.g., Indigenous groups) may not always be equipped to meet such demanding reporting requirements. Program administrators and users recommended providing standard reporting templates and/or training on interim and final reporting (e.g., activity and financial reports, records of decisions) as means to enhance recipient compliance with reporting requirements. Similar practices are already in place in other government departments (e.g., Environment and Climate Change Canada, Fisheries and Oceans Canada).

4.2.1.3 GCCP Post-Project Evaluation and Performance Reporting
Expectation:
  • Post-project evaluation is taking place.
Indicators:
  • % of projects with post-project evaluation forms.
  • Information contained in evaluations are not duplicating recipient performance reporting.

The RMAF/RBAF (2005) requires “monitoring authorities” (i.e., program administrators) to undertake an annual review and analysis of funding applications and final reports (including relevant evaluations, audits and client satisfaction surveys) to ensure that the GCCP approval processes are effective and timely and to assess progress towards results, challenges and lessons learned. The results of this review are to be reported in an annual report and used to assess the program’s design and delivery and, if needed, make the necessary adjustments. For example, data collected from program users on project results can be used to better inform risk assessments on future projects.

Project evaluations are still not being completed as required.

The previous GCCP evaluation (2010) found that relevant performance information was being inconsistently captured across agreements and was not centrally collected. To address this challenge, the evaluation recommended that the Agency develop a central data capture mechanism to ensure consistent and consolidated reporting. In 2013, the Centre of Expertise implemented a standard template for evaluating final contribution agreement performance (i.e., post-project evaluation). This evaluation form is expected to be completed by program users and submitted to the Centre of Expertise for central record keeping within 30 days of project completion. This requirement is clearly outlined on the Agency’s intranet as a distinct and final step in the GCCP Process.

In addition to basic tombstone information (e.g., project description and details on project cost, timeframe, activities and recipients), the post-project evaluation form also requires an assessment of:

  • project results and outcomes;
  • impact on beneficiaries;
  • recipient capacity to manage the project and deliver on reporting requirements; and
  • overall experience in relation to efficiency and cost-effectiveness.

We found these to be relevant performance indicators that do not duplicate information provided in recipient reports. However, the utility of the resulting information is not clear. The indicators are assessed by program users with Likert scale type questions (i.e., rating from 1 to 5). Responses are not required to integrate feedback from funding recipients. A similar evaluation form used in a department included in our comparative analysis offered examples of additional relevant indicators, including a number of open-ended questions to provide more details on how outputs and results relate to broader program objectives.

  • Additionally, a review of the program’s administrative database found that the evaluation forms were not systematically completed by program users. While the Centre of Expertise has made some recent efforts to increase the post-project evaluation form’s completion rates, of the 93 contribution agreements completed since the implementation of the form (i.e., from 2013-14 to 2014-15), data indicates that only 51% completed some form of post-project evaluation. The Centre of Expertise has just 33 of these completed evaluation forms in its central file. In other cases, it is possible that the form was never completed or that an email with an assessment of the project was sent instead.[20]
  • This finding was also confirmed by program users. In response to our evaluation survey, only 27% (n = 12) reported having completed an evaluation form upon completion of their project(s). A few survey respondents reported not even being aware of the form’s existence while others believed that the post-project evaluation form was not a useful tool.

As a result, the Centre of Expertise continues to lack the capacity to generate overall program-level analysis of the GCCP program’s performance. This was identified as an information gap by key programs administrators.

There is no annual performance reporting on the GCCP.

The RMAF/RBAF (2005) is clear on performance reporting expectations for the GCCP. These annual reports should contain a description of all projects, project recipients, project type, coordinates of recipients, and expended funds. It should also report on all performance indicators and document any exceptionally successful projects or accomplishments.

We found that the ambiguity and uncertainty surrounding who is responsible for this requirement of overall program reporting that was noted in the 2010 evaluation seems to have been clarified; program administrators were in agreement that this role should be fulfilled by the Centre of Expertise. Further, while all program administrators agreed that some form of program performance monitoring and reporting was needed, no definitive decision has been made with respect to the format and specific information it should contain. As a result, no annual review or annual reports have yet been completed and specific RMAF/RBAF requirements continue to be unmet.

A review of practices in other government departments seems to suggest that centralized annual (or periodic) reporting on performance and value for money is not consistently implemented for all G&C programs. For such reporting to be completed, it is a minimum requirement that performance data for individual agreements be systematically collected and maintained by program users and/or administrators. 

4.2.1.4 GCCP Data Records
Expectation:
  • Reliable records are maintained.
Indicators:
  • Evidence of quality assurance (QA)/quality control (QC) for GCCP administrative database.

As part of its program oversight role, the Centre of Expertise is responsible for monitoring compliance with the approved administrative and management standards/processes and tracking results achieved by the agreements to ensure value for money (in consultation with the relevant program officers/ managers). At a practical level, this translates, for example, into making sure that correct document templates are used, that all approved contribution agreements have obtained required signatures, demonstrate timely processes, and that the funds are disbursed in an appropriate and timely fashion.

Maintaining reliable information records of contribution agreements is critical to this monitoring role. To this end, the Centre of Expertise maintains an administrative database containing tracking data on all approved contribution agreements. As noted, in addition to project tombstone information, it contains information to track signature dates for critical steps in the approval process, information on approved and disbursed payments (linked and reconciled with the Agency’s financial system), and an indication of completed post-project evaluations.

Since the last GCCP evaluation (2010), the program’s administrative database was redesigned to better capture data that align with the Centre’s oversight role. Despite improvements, examination of the data recorded revealed that significant data integrity concerns persist. Data quality concerns include:

  • Financial information had not been updated for the past two fiscal years (2014-15 and 2015-16). While the Centre was able to update the data for the evaluation, it still contained a number of erroneous financial entries.
  • Information is missing for numerous agreements across various fields, including basic tombstone data (e.g., targeted program area, recipient type) and approval dates for critical process steps.
  • Fields contain erroneous information (e.g., recipient types did not align with categories in Ts&Cs, data in fields used to identify whether or not agreements were multi-year or met certain financial thresholds did not align with details of the actual agreements).
  • Abandoned agreements are not systematically flagged or deleted from the administrative database. The status of these agreements can only be deduced from the fact that no payments had been made.

Given these data issues and limitations, it is not clear how the Centre is able to effectively monitor compliance and track results achieved by all GCCP agreements.

We found that representatives of benchmark programs identified hyperlinking copies of key documents to the administrative database as an effective way to improve record maintenance practices. While Parks Canada’s Centre of Expertise is supposed to keep electronic copies of key documents on a central drive, these documents are not linked to the administrative database. Further, our file review found that some key documents were missing from the central file. For example, while nearly all files contained a record of CEO approval (either letter to recipient or CEO briefing note), only 77% (27 of 35) contained a copy of the original and/or amended contribution agreement. For the most part, the evaluation was able to obtain missing documents from respective program users or Chiefs of Finance and Administration.[21]

4.2.2. Outcomes

4.2.3.1 Alignment of Funded Projects with the GCCP’s Expected Results and Outcomes
Expectation:
  • Projects funded are consistent with GCCP expected results and outcomes (in Ts&Cs).
Indicators:
  • Number, type and value of agreements, by GCCP expected results or outcomes (in Ts&Cs).

The GCCP’s Terms and Conditions require that funded projects must support at least one of the ‘expected results and outcomes’ specified for the program. As noted, program theory indicates that achievement of these expected results will in turn assist the Agency in achieving its mandate. We evaluated the extent to which funded projects are aligned to achieve these results and outcomes.

As previously discussed (see section 4.2.1.1), while each project’s expected results are assessed by program users and confirmed by the Centre of Expertise during the initial phases of the GCCP Approval Process, these assessments are not systematically documented in project files. Further, projects’ alignment to the program’s expected results and outcomes is not explicitly tracked in the GCCP’s administrative database or referenced in individual agreements. As such, there are no available records that specify which of the program’s expected results and outcomes (as defined in the Ts&Cs) were set for each agreement.

For the purpose of this evaluation, the best proxy is a field in the GCCP administrative database that describes the “purpose of the project”. We used this data field to categorize funded GCCP projects by the program’s expected results and outcomes (see Table 8). Sufficient data was available in the database to categorize 273 (83%) of the 330 projects funded during the period covered by this evaluation (2010-11 to 2014-15).[22]

Table 8. Number and Value of Contribution Agreements for Projects Funded under the GCCP by Expected Results and Outcomes (2010-11 to 2014-15)
  Agreement Value ($000)
GCCP ‘Expected Results and Outcomes’ (2015) Number % Amount % Average Amount per Agreement
Note: Approximately 26% of contribution agreements had a stated purpose that aligned with two or more of the program’s ‘expected results and outcomes’. As a result, the total number and value of agreements in each column exceeds the actual totals for the fiscal years under evaluation.
1. Canadians recognize, appreciate and are engaged in the values of natural and cultural conservation 122 45% 22,520 50% 185
2. Stakeholders are engaged in terms of interest and involvement of common objectives towards ecological or cultural integrity 100 37% 16,466 36% 166
3. Parks Canada managers and stakeholders have access to a better knowledge base for informed decision making and dialogue on commercial, ecological or aboriginal issues of mutual interest 54 20% 7,010 15% 130
4. Visitors are provided opportunities to purchase souvenirs or experience traditional meals (e.g. Fortress of Louisburg) 10 4% 1,100 2% 110
5. Heritage assets are protected, secured and researched 7 3% 804 2% 115
6. Targeted audiences are educated in such areas as ecology, safety and other issues 65 24% 7,841 17% 120

Source: Analysis of GCCP Administrative Database (Dec. 2015).

Our analysis suggests that all categorized projects funded through GCCP supported at least one of the ‘expected results and outcomes’ described in program’s the Ts&Cs. The first two categories (i.e., those related to the engagement of Canadians and stakeholders) presented the highest proportion of funded projects both in number and total value. This is consistent with findings from our survey where similar proportions of programs users reported having managed a GCCP project that supported each of these results and outcomes.

Further, we noted a correlation with eligible recipients for two categories of expected results that suggests particular niches of interest. The majority (58%) of funded projects across all expected results and outcomes categories were led by non-profit organizations. However, most projects related to stakeholder engagement were managed by Indigenous groups or communities, with projects largely supporting their participation in consultation processes or in the development of cooperative management or impact benefit agreements. Similarly, most projects related to increasing the knowledge base were managed by members of the Canadian research community (i.e. universities), with projects largely supporting the monitoring of various ecosystem elements, summarizing research results and/or providing input for the development of related management strategies and procedures.

4.2.3.2 Achievement of Results Set Out in Contribution Agreements
Expectation:
  • Projects achieve or are likely to achieve the results set out in contribution agreements.
Indicators:
  • Extent to which agreements are carried out as intended (i.e., services and products delivered).
  • Results reported on contribution agreement post-project evaluation forms.
  • Perceptions of program users.

Given the data available to the evaluation, it is difficult to determine the extent to which contribution agreements funded under the GCCP are carried out as intended (i.e., services and products delivered). As noted, while program users are expected to maintain a record of results achieved, these are not available in a central file. Further, the evaluation form intended to indicate post-project results is either not being completed or not submitted to the Centre of Expertise for most contribution agreements.

Our evaluation reviewed a sample of 35 contribution agreements to determine the extent to which results set out in these agreements had been achieved. We found that all projects included in the sample were substantially carried out as intended at the time when the agreement came to term, i.e., most or all expected outputs were delivered as planned.[23] These included a number of services and products, such as: visitor experience products; research or information products; negotiation frameworks or planning documents, communications or media products; environmental interventions; meetings or conferences; and training courses. About 66% of files reviewed included some form of information on project outcomes based on indicators such as the reach of the project (e.g., event or conference attendance, media coverage, website visits) and/or the quality of outputs produced (e.g. surveyed satisfaction, impact of environmental interventions).[24]

While our sample was not representative, these results include the ten highest value contribution agreements (i.e., $500,000 and over) completed over this period suggesting that at minimum the most material projects are being completed as planned. There is also some evidence to suggest that these results can be generalized to the program as a whole; in responses to our survey, the vast majority of program users (93%, n = 38) reported that the GCCP contribution agreements that they were involved in achieved all or most expected results and outcomes.[25]

Further evidence of the performance of contribution agreements can be found in the limited number of completed GCCP evaluation forms (n = 33). According to reports from program users:

  • The majority (84%) of relevant projects were assessed as having achieved their expected results and outcomes.[26]
  • All projects (100%) were reported to have successfully reached their intended beneficiaries. While it is not clear how this is assessed, program users indicated that 83% of projects had received positive feedback from beneficiaries.
  • Nearly all recipients (97%) were reported to have demonstrated that they possessed the capacity to manage the funded projects. As a result, most (86%) of project users recommended working with the recipients on future contribution agreements.

4.2.3. Roles and Responsibilities

Expectation:
  • Program governance and accountabilities are clear and understood by all parties.
Indicators:
  • Documented roles, responsibilities and accountabilities (e.g., in process templates and contribution agreements).
  • Self-reports of awareness and understanding of roles and responsibilities.

As shown in Table 2, program accountabilities for the GCCP are shared across the Agency. The various roles and responsibilities pertaining to the planning, management and administration of the program are outlined in the program’s RMAF/RBAF (2005).

Overall, the majority of program administrators and users indicated that the corporate documents for the program such as the GCCP’s RMAF/RBAF (2005) and Ts&Cs (2015) provided them with a clear outline of their roles and responsibilities.[27] The Centre of Expertise has also developed a summary table of roles and responsibilities and relevant templates for the GCCP Approval Process which provides an easily understandable step-by-step outline. This summary table is accessible on the Agency’s Intranet. Further, we found that the Centre of Expertise is available to provide ad-hoc guidance to program users whenever clarifications are required.

Through our file review, we found that program users consistently make use of available templates (with the exception of the aforementioned evaluation form). These findings were also confirmed by program administrators.[28]

However, the evaluation found that in practice specific areas of accountability may remain unclear or challenging for some program users, programs administrators and recipients. These instances include:

  • Access to current Terms and Conditions (2015).
    • There was a significant delay in posting the most recent Ts&Cs on the Intranet (i.e., approved in May 2015 and posted in July 2016). As such, program users may have been referencing outdated documents to assess the alignment of potential projects with program eligibility criteria. Timely access to this key reference document is critical for proper program functioning.
  • Responsibility for Obtaining Appropriate Approvals Prior to Developing and Negotiating Agreements.
    • Evidence suggests that some program users may still have an unclear understanding of the GCCP approval process. We confirmed at least three cases where program users entered into a contribution agreement with recipients without obtaining the appropriate approvals. This may have been linked to a misunderstanding of the meaning of “pre-approval” in the recently implemented GCCP Annual Plan. In such cases, the Agency had to breach its own guidelines and obtain retroactive approvals. This creates a reputational risk for the Agency and can have potentially significant negative effects on the relationship with the recipient partner.
  • Responsibility for Reporting on Results.
    • As discussed, program administrators and users reported a lack of awareness of requirements for reporting on project outcomes and assessing value for money (i.e., post-project evaluations).
  • Strengthening the Role of Centre of Expertise.
    • The 2010 evaluation suggested a larger role for the Centre of Expertise to make the program more strategic (i.e., identifying areas for collaboration and minimizing duplication) and enable a centralized roll-up of results achieved with the GCCP agreements. Issues related to managing and monitoring the program’s overall implementation and success have largely remained unaddressed.

In response to our evaluation survey, program users indicated that providing additional GCCP training (or a detailed information package) would assist in further clarifying roles and responsibilities related to the management of GCCP agreements. Suggested training content logically included an overview of the general GCCP approval process as well as guidelines/best practices for monitoring agreements and evaluating/reporting on outcomes relating to effectiveness and value for money. This initiative was also suggested as a best practice by some of the GCCP program administrators. The evaluation found that other government departments who administer similar G&C programs have developed training materials as well as user manuals to better assist them in understanding roles and responsibilities.

4.2.4. Efficiency and Economy

A program is efficient to the extent a greater level of output is produced with the same level of input, or, a lower level of input is used to produce the same level of output. The level of input and output could increase or decrease in quantity, quality, or both. A program is economical to the extent the cost of resources used approximates the minimum amount needed to achieve expected outcomes.

One of the key purposes of the GCCP administrative database is tracking planned and actual expenditures for all contribution agreements to ensure that funds are disbursed as intended, in a timely fashion. We used this data to examine the extent to which funding for contribution agreements is being dispersed as planned. Analyses of this data should be interpreted in light of data integrity issues identified in section 4.2.1.4.

4.2.4.1 Dispersion of Planned Program Funding
Expectation:
  • Funding for contribution agreements is dispersed as planned.
Indicators:
  • Difference between planned and actual expenditures.
Description of Planned Allocation

Parks Canada’s GCCP administrative database shows that over the period covered by this evaluation (2010-11 to 2014-15) a yearly average of $8.2M was approved for GCCP expenditures, for a total of $41.1M. A summary of the yearly approved allocations is presented in Table 9.

Table 9. GCCP Approved Allocations ($000, 2010-11 to 2014-15)
  2010-11 2011-12 2012-13 2013-14 2014-15
Approved Amounts 7,208 9,884 12,475 6,094 5,478

Source: GCCP Database (Dec. 2015).

Description of Actual Expenditures

From 2010-11 to 2014-15, Parks Canada’s financial system (STAR) shows total expenditures for the program to be approximately $40.9M. This represents an average of $8.2M per year. Data in the GCCP’s administrative database indicates a total of $39.4M in expenditures related to specific agreements over this same period, i.e., $1.5M less than the amount reported in STAR (see Table 10).

Table 10. GCCP Expenditures ($000) by Source (2010-11 to 2014-15)
  2010-11 2011-12 2012-13 2013-14 2014-15
STAR Financial System[29]     
GCCP Database Total 7,033 9,678 12,130 4,670 5,846
National Office 3,232 5,305 9,850 3,251 2,985
Operational Units 4,794 4,455 2,361 2,082 2,589
Total 8,026 9,760 12,211 5,333 5,574

Source: STAR (March 2016) and GCCP Database (Dec. 2015).

Both data sets show that GCCP expenditures have decreased since their peak in 2012-13. This decrease may be largely attributed to one high materiality contribution agreement with the Trans Canada Trail Foundation (i.e., $10M for the period of 2010-11 to 2012-13). Specifically, a large disbursement of funding ($5.9M) occurred in 2012-13. PCA’s support to the TCT Foundation has since shifted to a Grant to support the Foundation’s fundraising campaign (see Appendix E).

As shown by data captured in STAR, the majority (60%) of expenditures were incurred for agreements managed by program users at National Office.[30] Among National Office expenditures, a high proportion was incurred by program users from the External Relations & Visitor Experience and the Protected Area Establishment and Conservation Directorates (5-year average of 50% and 38%, respectively).

Comparison of Planned and Actual Expenditures

In comparing the data on allocations (Table 9) and expenditures (Table 10), we found that both the Agency’s financial system and the GCCP’s administrative database show that overall program expenditures for most of the five year period are lower than the planned funding (i.e., amount approved in original or amended contribution agreements).

While data integrity issues make it difficult to reconcile numbers, there is reasonable evidence suggesting that the majority of funding for contribution agreements has been dispersed as planned. Of the 330 contribution agreements found in the database for 2010-11 to 2014-15, the majority (94%) presented a balanced or positive variance (i.e., actual expenditures were equal to or less than planned expenditures). This positive variance is to be expected in many cases as the total expenditures approved in each agreement is an estimation of the cost of the project (i.e., an “amount up–to”) and may not be required in full. Among completed agreements with a positive balance, over two-thirds had a balance of 30% or less of the approved expenditures, suggesting that the majority of approved funds were distributed. 

While 19 individual contribution agreements showed a negative variance in the database, we believe these discrepancies may mostly be explained by data integrity issues. For example, in some cases, payments appeared to not have been made in the correct fiscal years (i.e., one payment should have been made in 2011-12 but appears in the database for 2010-11). Some issues also arose where foreign currency was used for planned expenditures, but actual expenditures were expressed in Canadian currency. Findings from our file review also showed that in most cases where overpayments were indicated in the database, information contained in the file indicated correct payments had been made. Further, it is possible that amended information on planned expenditures may have not been updated in the database to reflect changes in payment schedule.

While the database contains evidence that financial monitoring is occurring (i.e., variances between approved and disbursed funding are calculated and files where there appears to be an issue are flagged), the extent of data integrity issues found lead us to question the effectiveness of these monitoring practices.

4.2.4.2 Timeliness of GCCP’s Administrative Processes
Expectation:
  • Administrative processes are timely and appropriate
  • Management has used available flexibilities to encourage efficient and economical operations.
Indicators:
  • Trend in process cycle times.
  • Steps in process are completed consistent with program’s notional timelines:
    • GCCP Annual Plan produced in advance of fiscal year;
    • final approvals of agreements within one month of the sign-off;
    • evaluation completed within 30 days of end of agreement.
  • Perceptions of program administrators and users.
  • Extent management has used available flexibilities to encourage efficient or economical operations.

The administrative process surrounding the approval of contribution agreement involves that responsible managers, in collaboration with the potential recipients, complete all documentation required for the proposed agreement and submit this for a series of reviews and approvals.[31] Currently, this requires each proposal to obtain six or seven levels of approvals, as detailed below.

Figure 4: Levels of Approvals for the GCCP Administrative Process

 

Figure 4: Levels of Approvals for the GCCP Administrative Process - Text Version
  • Originator
  • Director/Field Unit Superintendant and Vice-President (if not pre-approved)
  • Director, Procurement, Contracting and Contributions
  • National Director Corporate Communications
    (Review of Communication approach only)
  • Chief Financial Officer
  • Chief Executive Officer
  • Final approval
    (i.e., proceed with contribution agreement)

The previous GCCP Evaluation (2010) identified issues related to the timeliness of various steps of the project approval process. It recommended that the Agency review its approach to project approval and, at a minimum, propose internal service standards to clearly communicate expectations with respect to process timelines.[32] In its response, the Agency did not agree to develop service standards but rather indicated that timely processing will be addressed by:

  • Implementing an Annual Plan to ensure that CEO project approval for an agreement is in principle secured before the beginning of the fiscal year (see section 4.2.1.1).
  • Aiming to have all final approvals of contribution agreements secured within one month of the sign-off by the originating Director or Field Unit Superintendent.

A number of additional modifications to the approval process have also recently been implemented in an effort to improve its efficiency:

  • Development and/or continuous update of standardized templates:
    • Since 2011, the Centre of Expertise has developed and/or ensured the continuous update of a number of templates (i.e., one for each document to be completed as enumerated above) to ensure that contribution agreement proposals contain all necessary information and reduce burden of review.
  • Streamlining of the communication approach:
    • In November 2015, the Centre simplified the requirements for developing the communications approach for each contribution agreement (now all integrated into one document). This level of review is also now initiated earlier in the process in order to reduce delays in the final approval of contribution agreements.
  • Recipient letters no longer require CEO signature:
    • Since September 2015, letters informing recipients that their project has been approved are no longer required to be signed by the CEO. Once the agreement has received CEO’s approval, this letter may be signed by the relevant Director, Field Unit Superintendent or VP.

While data on the process cycle contained in the program’s administrative database suggest that these changes have enhanced the timeliness of the approval process, the objective to have all final agreement approvals secured within one-month following approval by the Director/Field Unit Superintendent has not yet been consistently met. Figure 5 shows the trend in the average number of calendar days between when the Centre received a signed-off proposal from the originating Director or Field Unit Superintendent and sign-off by CEO.[33] While the average has improved over time (from 68 days in 2010-11 to 35 days in 2014-15), in total only 33% of agreements were signed within 30 days.[34] Once CEO approval was obtained, it took on average another 43 calendar days to obtain all remaining signatures for the contribution agreement.[35]

Figure 5: Average Number of Calendar Days between Originating Director/FUS/VP and CEO Sign Offs (2010-11 to 2014-15)

 

Figure 5: Average Number of Calendar Days between Originating Director/FUS/VP and CEO Sign Offs (2010-11 to 2014-15) - Text Version

2010-11: 68
2011-12: 73
2012-13: 43
2013-14: 56
2014-15: 35

Source: GCCP Administrative Database (Dec. 2015).

Further, the previous evaluation (2010) recommended that the Agency should assess its level of risk tolerance and clearly communicate this in its rationale for any proposed revisions in its approach to approvals. The Agency responded that it would continue its current policy, i.e., that all contributions are to be approved by the CEO as per the Agency's Financial Delegation of Authorities authorized by the Minister. As a result, a majority of program administrators and users continue to report that the GCCP approval process remains slow, cumbersome and ill-suited to lower value agreements. Some respondents noted that the level of effort required to obtain approvals was resulting in missed opportunities. Similarly, post-project evaluation forms indicated that only about half (58%) of program users agreed that the GCCP process was efficient and cost-effective.

Suggestions provided to improve the timeliness of GCCP project approvals mostly support the need for the Agency to further implement a risk-based approach. Most importantly, program administrators and users recommended that the Agency scale the documentation and/or levels of approval required to the materiality of the agreement (e.g., CEO could delegate approval for low materiality agreements).

The notion of a risk-based approach is an integral concept of the TB Policy on Transfer Payments (2012). Results from the 2013 TBS assessment of G&C programs indicate that the majority (87%) of departments assessed calibrated application requirements based on the risk rating of applicants for most of their G&C programs and that 78% used a simplified application form for low-risk programs.

Delegation of authority for transfer payment programs such as the GCCP is permitted by the Financial Administrative Act (1985). This practice has already been successfully implemented in other federal departments. While the evaluation found that the Agency has recently put in place the mechanisms required to implement such delegation (e.g., provisions added to the Agency’s renewed Financial Delegation of Authority, revision of GCCP Ts&Cs), it has not yet implemented any change in practice to its delegation of authority for contributions.

Other complementary suggestions for improving the efficiency of the GCCP approval process included starting the annual planning process earlier in the year (prior to beginning of calendar year) or obtaining CEO’s final approval in the GCCP Annual Plan. These were suggested as a way to enable recipients to secure and receive funding at a more opportune time. With projects being pre-approved by the CEO in the GCCP Annual Plan in March or April, the final approval, signature of the agreement and dispersion of funds often occurred well into the new fiscal year, often too late for recipients to start timely project planning.

4.2.4.3 GCCP Administrative Costs
Expectation:
  • The costs of producing outputs and results are less than or the same as costs of comparable benchmarks.
Indicators:
  • Ratio of administrative costs to total funds dispersed (as compared to benchmarks). 

As responsibility for program administration is dispersed across the Agency, the administration costs of the program (financial and human resources) are not consistently tracked. According to the program’s RMAF/RBAF (2005) the costs to the Agency of managing, administering and auditing/evaluating the GCCP are estimated to be approximately $300,000 per year.[36] These costs are all associated with percentages of employees’ time. The previous GCCP evaluation (2010) found that this estimate was flawed, as they were:

  • based on an outdated estimation of related expenditures, i.e., referenced to $2M per year in expected agreement expenditures rather than current average ($8.2M from 2010-11 to 2014-15); and
  • represented mostly costs to be incurred by National Office, which may under-represent costs associated with management of agreements at the local level.

This finding lead to the recommendation that the Agency provide a reasonable basis for determining the administrative costs of the program in order to be able to assess program efficiency. The Agency disagreed; it estimated that the costs of gathering and reporting on the administrative costs of the GCCP would outweigh the benefits that could be derived from the information. Given this, the current evaluation could not obtain an accurate estimate of administrative costs for the program.

The evaluation found that other government departments managing “class-type” G&C programs do not systematically track administrative costs due to the decentralized nature of these programs. Where estimated, administrative costs were roughly assessed to represent 1.5% or less of total program funding expenditures. This benchmark is consistent with current Parks Canada estimates.

While the Centre of Expertise has faced some recent staffing challenges, with staff turnover in both related positions, no significant gaps were found with the program’s current human resource allocation. Program administrators mentioned that the resources allocated for the Centre of Expertise (i.e., two positions partly dedicated to the program, i.e., one manager and one business analyst) were sufficient to effectively carry-out the current level of activities. However, it was noted that any expansion in the role played by the Centre (such as increased performance reporting) may require further assessment of necessary resources.

5. Conclusions and Recommendations

Overall, we found that the GCCP is relevant and consistent with the priorities, roles and responsibilities of both the Parks Canada Agency and the Government of Canada. A continued need for the program is justified by the ongoing demand for the program and by the absence of other partnering/funding mechanisms within PCA that offer sufficient flexibility to meet the objectives of funded projects. Further, evidence suggests that GCCP funding enables projects that otherwise may not have taken place. However, the evaluation identified opportunities to increase the relevance of the program by broadening specific requirements outlined in the GCCP’s Terms and Conditions to better address the Agency’s ongoing and emerging corporate priorities.

In addition, we found that for the most part, the prescribed process for entering into and carrying out a contribution agreement under the GCCP is being followed, and relevant funds are being dispersed. Approved and funded projects are aligned with expected requirements detailed in GCCP Ts&Cs, are generally implemented as intended and produce expected outputs. However, a number of areas for improvement were identified, particularly related to program monitoring and reporting. The Centre of Expertise’s roles and responsibilities for ensuring ongoing reporting on agreement-level results (i.e., outcomes and value for money) and for ensuring a strategic and centralized functioning of the program are not consistently being fulfilled. Specifically, we found that:

Project Eligibility Assessment and Approval:

  • Project eligibility assessments are being completed. However, there is no standardized tool/format used for these assessments and results are not documented by program users as required.
  • Project approvals are taking place, including pre-approvals (both within and outside the GCCP Annual plan) and final approvals.

Agreement-Level Monitoring and Reporting:

  • Program expectations for activity and performance monitoring and reporting have been established and are reflected in the final agreements. While evidence indicates that responsible program users are involved in monitoring projects, we found that recipient reporting requirements (e.g., final activity and financial reports) are not being consistently produced and/or records of such requirements are not consistently maintained by responsible program users. 
  • Post-project evaluations which provide an assessment of results and outcomes of the agreements are inconsistently completed by program users.

Program-Level Monitoring and Reporting:

  • Program-level record maintenance practices present significant limitations. Data integrity issues were found with both the GCCP’s administrative database and individual agreement records. These data issues impede the Centre of Expertise’s ability to effectively monitor compliance and track the progress achieved by GCCP agreements or to assess the GCCP’s performance at the program-level.
  • The Centre of Expertise has yet to produce expected annual performance reports on the GCCP.

We also examined the extent to which the GCCP is efficient and economical. We found that for the five-year period under review (2010-11 to 2014-15), overall GCCP expenditures were reasonably aligned with approved allocations. Evidence suggests that the majority of funding for contribution agreements has been dispersed as planned. The administrative costs of the GCCP are not tracked; we found this to be a common practice for similar government contributions programs. The estimated costs represent a reasonable proportion of the program’s total expenditures as compared to available benchmarks.

The program has made some efforts to improve the efficiency of GCCP process by implementing a GCCP Annual Plan to obtain CEO pre-approval for upcoming projects within the year and continuously updating, clarifying and simplifying requirements and templates for the GCCP. However, we found that the Agency is still lacking a standardized risk-based approach for assessing project proposals to ensure that administrative requirements are aligned with the level of risk and materiality of individual projects. This approach was identified as a best practice by program administrators and users as well as by other government departments. As a result, key components of the GCCP process (e.g., approval structure, establishment of monitoring and reporting requirements) continue to be cumbersome and ill-suited to lower value agreements.

Recommendations

Key issues identified through this evaluation largely pertain to the administration of the program and contribution agreements. As such, our recommendations focus on the program’s administrative processes and requirements.

In light of opportunities identified to increase the relevance of the GCCP and enable the program to better address the Agency’s ongoing and emerging corporate priorities by broadening specific requirements outlined in the GCCP’s Terms and Conditions, we recommend that:

Recommendation 1:

The Agency’s Chief Financial Officer should review the GCCP’s Terms and Conditions to ensure that stated requirements, particularly as they relate to expected results and outcomes, are aligned with Parks Canada’s Departmental Results Framework (to be approved by November 2017). 

Management Response:

Agreed. The Chief Financial Officer Directorate will review the GCCP’s Terms and Conditions to ensure alignment with Parks Canada’s Departmental Results Framework. Should changes be deemed pertinent, revised Terms and Conditions will be submitted to Treasury Board for approval. Target: 2017-18.

The notion of a risk-based approach is an integral concept of the TB Policy on Transfer Payments (2012). Treasury Board encourages administrative and reporting requirements for applicants and recipients to be aligned with associated risks. This scaling of effort increases efficiency, enabling more attention and resources to be directed to areas of highest risk. We found that the Agency has yet to consistently assess project risks or scale its approach accordingly. Given that low materiality projects (under $50K) represent a high proportion of agreements funded under the GCCP, we recommend that:

Recommendation 2:

The Chief Financial Officer should implement a risk-based approach to the GCCP, putting measures in place to ensure that program requirements are aligned with the level of risk of the project being funded. This includes:

  • Developing and applying a standardized risk assessment tool for proposals, with clearly documented rating criteria; and
  • Reviewing the GCCP’s contribution agreement process to streamline processes (e.g., approvals, monitoring and reporting requirements) where risk is low.
Management Response:

Agreed. The Chief Financial Officer Directorate will implement a risk-based approach to the GCCP, including a standardized risk assessment tool and streamlined processes for low-risk projects. Target: 2017-18.

The Centre of Expertise’s capacity to ensure an oversight role for the program is significantly limited by various program data availability and data integrity issues at both the agreement-level and program-level. Given this, we recommend that:

Recommendation 3:

The Chief Financial Officer should review program-level performance monitoring and reporting needs and processes to ensure that: 1) they are aligned with the Centre of Expertise’s roles and responsibilities as they relate to ensuring the strategic and centralized functioning of the program; 2) they support a risk-based approach; and 3) that effective mechanisms to collect the required data are in place. As part of this review, the Chief Financial Officer should:

  • Consider the development and delivery of training courses/materials to ensure all programs users and recipients are made aware of GCCP requirements.
  • Develop mechanisms to ensure consistent recipient reporting (e.g., standardized templates).
  • Review the structure of current post-project evaluations and ensure that requirements to complete these evaluations are consistently met by program users.
  • Implement further quality assurance and quality control mechanisms to ensure that reliable program information is collected and monitored through the GCCP’s centralized administrative database and records. 
Management Response:

Agreed. The Chief Financial Officer Directorate will conduct a review of program-level performance monitoring and reporting needs and processes to identify necessary changes. Targets: Review completed, 2016-17; Changes implemented, 2017-18.

Appendix A: PCA Strategic Outcome and Program Alignment Architecture

 Strategic Outcome and Program Alignment Architecture

Strategic Outcome and Program Alignment Architecture - Text Version

Canadians have a strong sense of connection to their national parks, national historic sites, heritage canals and national marine conservation areas and these protected places are experienced in ways that leave them unimpaired for the enjoyment of present and future generations.

  • Heritage Places Establishment
    • National Park Establishment and Expansion
    • National Historic Site Designations
    • National Marine Conservation Area Establishment
    • Other Heritage Places Designations
  • Heritage Places Conservation
    • National Park Conservation
    • National Urban Park Conservation
    • National Marine Area Conservation
    • National Historic Site Conservation
    • Other Heritage Places Conservation
  • Heritage Places Promotion and Public Support
    • Heritage Places Promotion
    • Partnering and Participation
  • Visitor Experience
    • National Park Visitor Experience
    • National Urban Park Visitor Experience
    • National Marine Conservation Area Visitor Experience
    • National Historic Site Visitor Experience
    • Heritage Canal Visitor Experience
  • Heritage Canals, Highways and Townsites Management
    • Townsite Management
    • Highway Management
    • Heritage Canal Management
  • Internal Services
    • Governance and Management Support
    • Resource Management Services
    • Asset Management Services

 

The GCCP is not represented as a specific activity or sub-activity in the Agency’s Program Alignment Architecture (PAA). Rather, it contributes to a broad scope of programs and sub-programs across the PAA.

Appendix B: Evaluation Matrix

A. Relevance

1. To what extent is there a continued need for the program?

Specific Questions
  • To what extent is there a continued need for the program?
  • To what extent is the program responsive to the needs of managers?
Expectations
  • The program addresses a continued need.
  • Managers use the GCCP.
  • The GCCP does not substantively duplicate other G&C programs.
Indicators
  • Trends in contribution agreements (i.e., number, value, etc.).
  • # of program managers using the GCCP.
  • GCCP contribution funding as % of total project value (to be extrapolated from file review).
  • Program administrators and users’ perception of need for GCCP.
Data Sources/Methods
  • Document review
  • File review
  • Interviews
  • Survey

2. To what extent is the program aligned with government and Agency priorities?

Specific Questions
  • To what extent is the program aligned with federal government and Agency priorities?
Expectations
  • GCCP Terms and Conditions[37] align with Government of Canada and PCA priorities.
Indicators
  • Degree to which GCCP Terms and Conditions align with GOC Whole of Government Framework.
  • Degree to which GCCP Terms and Conditions align with PCA priorities.
Data Sources/Methods
  • Document review

3. To what extent is the program aligned with federal roles and responsibilities?

Specific Questions
  • To what extent is the program aligned with PCA roles and responsibilities?
Expectations
  • The program is clearly aligned with PCA’s legislative and policy mandate. 
Indicators
  • Federal legislation, policies and directives indicate relevant roles and responsibilities.
  • PCA mandate, policies and directives indicate relevant roles and responsibilities.
Data Sources/Methods
  • Document review

B. Performance

4. To what extent are expected activities/ outputs being delivered?

Specific Questions
  • To what extent have the program’s intended activities and outputs been delivered?
Expectations
  • Applications are being reviewed and approved.
  • Contribution agreements for relevant projects are signed with eligible recipients.
  • Project monitoring is taking place.
  • Post-project evaluation is taking place.
  • Reliable records are maintained.
Indicators
  • Evidence of project qualification.
  • CEO Pre-Approval (GCCP Annual Plan and briefing notes).
  • Evidence of other approvals (signatures).
  • Number of contribution agreements.
  • Evidence of project monitoring activities by program users.
  • Evidence of performance information collected (monitoring of results).
  • % of projects with evaluation forms.
  • Information contained in evaluations are not duplicating recipient performance reporting.
  • Evidence of QA/QC for GCCP Database.
Data Sources/Methods
  • Document review
  • File review
  • Interviews
  • Survey

5. To what extent are expected outcomes being met?

Specific Questions
  • To what extent have funded projects contributed to the achievement of Agency objectives?
Expectations
  • <
  • Projects funded are consistent with GCCP expected results and outcomes (T&Cs).
  • Projects achieve or are likely to achieve the results set out in contribution agreements.
Indicators
  • Number, type and value of agreements, by GCCP expected result or outcome.
  • Extent to which agreements are carried out as intended (i.e., services and products delivered).
  • Results reported on contribution agreement evaluation form.
  • Perceptions of program users.
Data Sources/Methods
  • Document review
  • File review
  • Interviews
  • Survey

6. To what extent are roles, responsibilities and accountabilities for program delivery clear and effective?

Specific Questions
  • To what extent are roles, responsibilities and accountabilities for program delivery clear and effective?
Expectations
  • Program governance and accountabilities are clear and understood by all parties.
Indicators
  • Documented roles, responsibilities and accountabilities (e.g., in process templates and contribution agreements).
  • Self-reports of awareness and understanding of roles and responsibilities.
Data Sources/Methods
  • Document review
  • Key informant interviews
  • Survey

C. Efficiency and Economy

7. To what extent is the program efficient and economical?

Specific Questions
  • Are administrative processes timely and appropriate?
  • Are funds for contribution agreements dispersed as planned?
  • How do costs compare against benchmarks?
  • What management flexibilities constraints influence the program’s efficiency economy?
Expectations
  • Administrative processes are timely and appropriate.
  • Funding for contribution agreements is dispersed as planned.
  • The costs of producing outputs and results are less than or the same as costs of comparable benchmarks.
  • Management has used available flexibilities to encourage efficient and economical operations.
Indicators
  • Trend in process cycle times.
  • Steps in process are completed consistent with program’s notional timelines:
  • GCCP annual plan produced in advance of fiscal year;
  • final approvals of agreements within one month of the sign-off;
  • evaluation completed within 30 days of end of agreement.
  • Perceptions of program administrators and users.
  • Difference between planned and actual expenditures.
  • Ratio of administrative costs to total funds dispersed (as compared to benchmarks).
  • Extent management has used available flexibilities to encourage efficient or economical operations.
Data Sources/Methods
  • File review
  • Database analysis
  • Comparative analysis
  • Interviews
  • Survey 

Appendix C: Key Documents Consulted

Legislation 

  • (1998) Parks Canada Agency Act
  • (1985) Financial Administration Act

Parks Canada Documents

  • (1994) Guiding Principles and Operational Policies
  • (2004) Grants and Contributions Audit Final Report
  • (2010) Evaluation of Parks Canada’s General Class Contribution Program (and Management Action Plan)
  • (2016-17) National Historic Site Cost-Sharing Program Guidelines
  • (2010-11 to 2016-17) Report on Plans and Priorities (RPP)

GCCP Program-Related Documents

  • (2005) Integrated RMAF/RBAF for Parks Canada’s GCCP
  • (2011 and 2015) GCCP Terms and Conditions
  • (2010-11 to 2014-15) GCCP Annual Plans
  • (2015) Contribution Agreement Approval Process under the GCCP

Treasury Board Secretariat Documents

  • (2006) From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs
  • (2008) Government of Canada Action Plan to Reform Administration of Grants and Contributions,
  • (2008) Directive on Transfer Payments
  • (2009) Policy on Evaluation (as well as Directive and Guidelines)
  • (2012) Policy on Transfer Payments
  • (2013) Assessment of the impact of departmental implementation of “The Government of Canada Action Plan to Reform the Administration of Grant and Contribution Programs” 2013 Results Report
  • (2010-11 to 2014-15) Estimates Parts I and II, The Government Expenditure Plan and Main Estimates
  • (2015) Whole of Government Framework
  • (2016) Policy on Results (as well as Directive and Guidelines)

Other Government Department Documents

  • (2008) Audit of Class Grants and Contributions Program –Fisheries and Oceans
  • (2008) Evaluation of the “Class” Grants and Contributions Program –Fisheries and Oceans
  • (2009) Evaluation of Class Grants and Contributions – Environment Canada
  • (2011) Horizontal Audit of Transfer Payments Project – Natural Resources Canada
  • (2012) Evaluation of the Class Grants and Contributions Program – Natural Resources Canada
  • (2012) Report of the Auditor General of Canada: Grant and Contribution Program Reforms
  • (2013) Audit of Grants and Contributions – Environment Canada.
  • (2013) Evaluation of the “Class” Grants and Contributions Program –Fisheries and Oceans

Appendix D: Evaluation of the Grant To International Peace Garden

Program Description

The International Peace Garden (IPG) Inc. is a non-profit organization established in 1930 whose purpose is to create and maintain a garden as a memorial to the peace between Canada and the United States. The dedication of the garden took place in 1932. About 62% of the 2339.3 acre garden is located in Manitoba (1451.3 acres) and 38% in North Dakota (888 acres). 

The IPG is operated by a sixteen-person Board of Directors, of whom half are American and half Canadian. The Governor of North Dakota and Premier of Manitoba each appoint one director. The State and Province also provide equal funding to the garden, with some additional assistance provided by the U.S. and Canadian federal governments (primarily for capital projects).

From 1930 to 2001, the Government of Canada provided an annual, unconditional grant to the IPG (value ranging over time from $15,000 to $30,000). Following an internal audit of the grant (2001), a decision was made to require a formal request (i.e., an “application”) each year from the IPG prior to payment. The last grant agreement was signed in FY 2011-12 (valid until 2015-16). Each year, a letter from the International Programs Manager is prepared for the recipient to accompany a cheque for funding. Funds provided through Parks Canada are intended for general operations and maintenance of the IPG.

Evaluation Results

Continued Need for Program

Indicators

  • Program continues to address a demonstrable need.
  • Program is responsive to the needs of Canadians.

What We Found

  • The goal of the grant is to support the maintenance of the Garden as a memorial to the peace that has existed between the United States and Canada. As found by a previous evaluation (2010), the small size of the grant in relation to the IPG’s overall revenue suggests that PCA’s contribution is more of a symbolic nature than essential for the operation of the garden.
  • The garden is open year-round for Canadians to visit and learn about Canadian-US history.
Alignment with Government Priorities

Indicators

  • Alignment with the Whole of Government Framework.

What We Found

  • This grant is aligned with the Whole of Government Framework’s strategic outcome related to “A vibrant Canadian cultures and heritage” as it “support[s] Canadian culture and enhances knowledge of Canada’s history and heritage”.

Indicators

  • Alignment with Parks Canada strategic outcomes and/or PAA.

What We Found

  • This grant aligns with PCA’s expected results related to the Agency’s Other Heritage Places Conservation sub-program, i.e., that Parks Canada programs support the conservation of places administered by others.
Alignment with Federal Roles and Responsibilities

Indicators

  • Identification of legal authority for grant program.

What We Found

  • The legal authority for this grant program derives from Section 19(1) of the Parks Canada Agency Act.

Indicators

  • Identification of administrative authority for grant program (e.g., TB Submission, CEO approval, etc.).

What We Found

  • The Chief Executive Officer is the administrative authority for this grant.

Performance

Effectiveness

Indicators

  • Assessment of delivery of expected outputs (i.e., grant was/was not dispersed as planned).

What We Found

  • Planned funding for this grant entails that $22,700 be provided each year through Parks Canada (2010-11 to 2014-15; five year total of $113,500).
  • PCA financial system STAR shows that payments were made yearly from FY 2010-11 to 2014-15 in the amount of $22,700.

Indicators

  • Assessment of progress towards expected outcomes (i.e., some description of how funds granted are intended to be used).

What We Found

  • Funds are intended for general operations and maintenance of the gardens. The garden has maintained its operations over the period of time covered by this evaluation.

Indicators

  • Description of performance monitoring/reporting, if any.

What We Found

  • Since at least 2007, the Agency has required that the IPG provide annual audited financial statements. No other reporting is required.
  • We found that while the responsible PCA representative did not have the required audited financial statements on file, these were provided by the recipient upon request (on a calendar year basis).
Efficiency & Economy

Indicators

  • Description of expenditures (e.g., planned vs. actual, trend).

What We Found

  • Funds for this grant were disbursed as planned ($22,700 per year) from 2010-11 to 2014-15.

Indicators

  • Value of grant relative to total revenues and expenditures of recipient.

What We Found

  • PCA’s contribution to the IPG represents approximately 1.3% of total IPG revenue and 1.2% of total expenses.

Indicators

  • Value of grant relative to other sources of revenue of recipient.

What We Found

  • In most years, PCA’s contribution to the IPG represents 100% of Canadian federal funding and 2% of all other grant funding (i.e., state funding). Exception was in 2011, when additional funding was received from the Canadian government for specific programming initiatives. As a result, in this year, the grant represented just 8% of all Canadian federal government funding.

APPENDIX E: EVALUATION OF GRANT TO SUPPORT THE TRANS CANADA TRAIL FOUNDATION’S FUNDRAISING CAMPAIGN

Program Description

The Trans Canada Trail (i.e., The Great Trail) is a network of recreational trails that wind through every province and territory from coast to coast. The project was initiated in 1992 to celebrate Canada’s 125th anniversary. As of the end of 2014-15, an estimated 18,000 km (80%) of the Trail was already operational. When completed, The Trail is expected to be 24,000 kilometres long.

The initiative is directed by the Trans Canada Trail (TCT) Foundation, which is led by a sixteen-person Board of Directors forming six Standing Committees.38 It is being built, operated and maintained by community, regional and provincial/territorial trail groups, as well as by all levels of government.

In March 2014, Parks Canada signed a $25,000,000 grant with TCT Foundation to support their efforts to raise funds to complete the Trail by 2017, for the celebration of Canada’s 150th anniversary. The agreement involves a matching fund where Parks Canada will pay the TCT Foundation one dollar for every two dollars of actual pledges and cash donations received from non-federal donors from July 1, 2011 (retroactively) to March 31, 2017.

Evaluation Results

Relevance

Continued Need for Program

Indicators

  • Program continues to address a demonstrable need.
  • Program is responsive to the needs of Canadians.

What We Found

  • The Trail is to be completed by 2017. As of the end of 2014-15, over 20 % of the Trail remained to be built (6 000 km).
  • The purpose of the Trail is: “to draw Canadians closer to each other and to Canada’s unique, unrivalled and irreplaceable outdoor spaces where Canadians can connect to Canada’s natural and cultural heritage”. Four out of five Canadians live within 30 minutes of the Trail.
Alignment with Government Priorities

Indicators

  • Alignment with the Whole of Government Framework.

What We Found

  • This grant is aligned with the Whole of Government Framework’s strategic outcome related to “A vibrant Canadian culture and heritage” as it “support[s] Canadian culture and enhances knowledge of Canada’s history and heritage”.

Indicators

  • Alignment with other federal agreements/priorities.

What We Found

  • This grant is aligned with federal government commitments made by the Prime Minister in 2014 to support the achievement of the Trail for the 150th anniversary of Confederation in 2017.

Indicators

  • Alignment with Parks Canada strategic outcomes and/or PAA.

What We Found

  • This grant is indirectly aligned with PCA’s expected results for the Heritage Places Promotion sub-program. The TCT project is expected to contribute to Trail users’ awareness of national parks and national historic sites administered by PCA, through which the Trail passes.
Harmonisations avec les rôles et les responsabilités du gouvernement fédéral

Indicators

  • Identification of legal authority for grant program.

What We Found

  • The legal authority for this grant program derives from Section 19(1) of the Parks Canada Agency Act.

Indicators

  • Identification of administrative authority for grant program (e.g., TB Submission, CEO approval, etc.).

What We Found

  • The Chief Executive Officer is the administrative authority for this grant.

Performance

Effectiveness

Indicators

  • Assessment of delivery of expected outputs (i.e., grant was/was not dispersed as planned)

What We Found

  • Planned funding for this grant entails one dollar for every two dollars of actual pledges and cash donations received from non-federal donors for up to $25M.
  • Data collected for the evaluation confirms that the grant was dispersed as planned (see Efficiency & Economy, below).

Indicators

  • Assessment of progress towards expected outcomes (i.e., some description of how funds granted are intended to be used).

What We Found

  • In 2013-14, the focus of the TCT Foundation was on trail planning. As such, only $2.4M was disbursed out of the $5.5M in project funding that was approved. Another 110 kilometres were added to the Trail.
  • In 2014-2015, the TCT Foundation approved over $8M of private and public funding in trail development grants. This was dispersed to 58 trail-building organizations and municipalities, resulting in another 1,291 km of trail added to the network.
  • Overall, from 2013-14 to 2014-15, the Trail’s construction advanced 11% toward its final objective (i.e., from 69% to 80% completion).

Indicators

  • Description of performance monitoring/reporting, if any.

What We Found

  • Given that the grant is designed to match donations, we expected that the TCT Foundation provide the Agency with some proof of donations received. To this end, the provision of this grant require that TCT provide annual audited financial statements and proof of donations every six months.
  • These documents were provided to the evaluation by the responsible PCA representative.
Efficiency & Economy

Indicators

  • Description of expenditures (e.g., planned vs. actual, trend).

What We Found

  • The Main Estimates list Parks Canada’s expected expenditure on the grant to the TCT at $6.25M per year, starting in 2014-15 (up to $25M over three years).[39]. Over the two fiscal years covered by the evaluation, the total dispersed was $12.67M (see table). This leaves up to $12.33M to be dispersed by March 31, 2017.
Fiscal Year Financial Summary ($M)
TCT Donations Received Expected PCA Matching Donation Actual PCA Payments
2013-14 14,30[40] 7,15 7,15
2014-15 11,04 5,52 5,52
Source : TCT Financial Statements Star

Indicators

  • Value of grant relative to total revenues and expenditures of recipient.

What We Found

  • PCA’s contribution to the TCT Foundation represented 54% of the Foundations’ total revenues and 78% of their total expenditures from 2013-14 and 2014-15. This includes retroactive donations since July 2011.

Indicators

  • Value of grant relative to other sources of revenue of recipient.

What We Found

  • PCA’s contribution to the TCT Foundation represented 87% of the Foundation’s grant revenue from 2013-14 and 2014-15.

Footnotes

[1] The other contribution program administered by PCA is the National Historic Sites of Canada Cost-Sharing Program (NHSCCSP). The Agency is also a participant in the Aboriginal Economic Development Strategic Partnerships Initiative (AEDSPI) administered by Indigenous and Northern Affairs Canada.

[2] This is a new process implemented in 2015. Prior to this, the CEO signed the letter to the recipient.

[3] Explicit requirements for CEO approval were removed from the GCCP’s Ts&Cs in 2013-14. Contributions exceeding this amount continue to require Treasury Board approval.

[4] While responsibilities are frequently delegated to an operational level manager or officer, the senior officer retains ultimate responsibility.

[5] PCA does not directly produce but receives a variety of outputs as a result of recipient activities (e.g., products or activities for visitors, ecological integrity or commemorative integrity research, capacity and relationship building, support for designations or conservation, etc.).

[6] Four financial files appear to have been lost or misplaced while being sent from the centralized records facility.

[7] Additional funding was provided by the recipient and/or another organization.

[8] This count may be slightly inflated. A small number of projects may be the continuation of previous projects (renewed on a yearly basis rather than using a multi-year agreement).

[9] There is some overlap in this data for projects funded in 2010-11.

[10] The estimated value was not provided for three agreements where in-kind contributions were noted. Actual value of in-kind expenditures were only tracked for 2011-12.

[11] Parks Canada’s planned contribution to the Government of Canada’s celebration of the 150th anniversary of Confederation. This is a corporate priority in the Agency’s 2016-17 Report on Plans and Priorities (RPP).

[12] Percentages are calculated based on a total of 319 contribution agreements and a total value of approximately $43M. Database entries for a small number of files contained incorrect information (1%, n = 3) or missing information (2%, n = 8) and were thus excluded from this analysis.

[13] The previous GCCP evaluation (2010) had highlighted the need for expanding the list of eligible recipients to include for-profit organizations.

[14] Approval dates were not provided in the database for four agreements. Proof of CEO approval for one of these four agreements was found in electronic files kept by the Centre.

[15] Data explaining reasons for approved projects not progressing to a signed agreement is not available. Program users indicated that these reasons may include loss of interest by recipient, project non-feasibility, project delays, etc.

[16] Recipient categories found in the GCCP administrative database do not fully align with the categories outlined in the Ts&Cs; the database shows no reported contributions made to voluntary associations and Management Boards. However, closer examination of the files reveals that some recipients could be more accurately described as fitting into one of these categories and so these recipient types are depicted jointly in the table. To date, there have been no contribution agreements signed with for profit organizations or foreign governments.

[17] The program’s Ts&Cs (2015) include a number of project types that were not operational during the period under review. These are not listed in this table.

[18] This obligation stems from findings of a 2004 audit of the GCCP conducted by the Agency. The audit recommended that all contribution agreements contain clear and standard interim and final reporting requirements and that all funding agreement recipients be required to provide some form of final financial report.

[19] Treasury Board Secretariat. Assessment of the impact of departmental implementation of “The Government of Canada Action Plan to Reform the Administration of Grant and Contribution Programs”, 2013.

[20] The GCCP administrative database indicates that 30% of post-project evaluations were sent in an email. The evaluation found a few of these emails on file. 

[21] From all sources, we were not provided with the final CEO approval documents for two of 35 files (although one contained the CEO approval for amendment) and the signed contribution agreement for one case.

[22] The categorization was completed with the limited information provided and may not accurately represent the full scope of the projects. Single projects could be linked to multiple expected results and outcomes.

[23] About 34% of agreements (n = 12) were amended to either extend project duration, increase project funding, change reporting requirements and schedules or change payment schedules within the initial allocated timeframe. This enabled the recipients to adjust to unforeseen delays and events and continue to carry out the projects without modifying their intended core nature. Even with these flexibilities, some projects were still in progress at the close of agreement (e.g., completion of a thesis, meetings toward negotiation of an agreement framework). For two agreements, some reporting requirements and outputs were delivered at the end of a subsequent agreement rather than during the timeframe of the initial agreement. 

[24] Agreements of <$100K are only required to report on outputs; these do not necessarily have results information at the outcome level. Performance measurement provisions in some of the high value files (i.e., ≥$100K) were also limited to outputs.

[25] Three respondents indicated that challenges related to the complexity of measuring outcomes for some of the GCCP contribution agreements they were involved in limits their ability to conclusively report that all expected outcomes were achieved (e.g., complexity of measuring outcomes related to promotion activities, challenges associated with Indigenous recipients’ limited capacity for measuring and reporting on performance requirements).

[26] Contribution agreements which were reported to have experienced challenges in achieving expected results and outcomes pertained to projects that involved the negotiation of an agreement framework with Indigenous groups and communities (i.e., a complex process that requires extended periods of time).

[27] The previous evaluation (2010) found that contribution recipients reported having a clear understanding of their roles and responsibilities. No significant issues were found for this evaluation.

[28] Program administrators noted the use of outdated templates by some of the more frequent program users. However, this was said to be easily rectified during the review process completed by the Centre.

[29] These amounts match those reported in the Agency’s Public Accounts.

[30] This trend was consistent across fiscal years, with the exception of 2010-11 where operational units were responsible for the majority of GCCP expenditures (60%).

[31] This includes six documents: contribution agreement; CEO briefing note; letter to recipient; communication approach; information form for proactive disclosure; and transmittal slip for tracking of all approval signatures.

[32] Under the TB Policy on Transfer Payments (2012), deputy heads are required to establish reasonable and practical departmental service standards for transfer payment programs. Service standards state a level of performance that citizens, recipients or users can expect of a program and typically focus on the timeliness of delivery.

[33] Data gaps exist in administrative database; about 17% of agreements lacked valid data on approval dates. A higher percentage of missing data found for the last fiscal year under review (2014-15) may impact on the accuracy of these results.

[34] There was significant variability in the time lag to obtain signatures (from one day to a maximum of 423 calendar days).

[35] There is less variability is this last step; almost half (46%) of agreements were signed within 20 calendar days.

[36] Break-down: Finance Branch - $125K; Audit and Evaluation - $75K; other administration - $100K.

[37] Specifically, we would expect the GCCP’s stated objectives, expected results and eligible projects to align with GOC and PCA priorities.

[38] There are no federal, provincial/territorial or municipal government members appointed to the Board.

[39] Grant agreement was signed in March 2014.

[40] This amount includes retroactive donations since July 2011.

Date modified :