Evaluation by Leeds Beckett University June 2016

 

SUSTAINABLE SUNDERLAND PROGRAMME EVALUATION

Dr James Woodall, Jenny Woodward & Susan Coan
June, 2016

 

Context

Sustainable Sunderland is a programme which aims to increase awareness and understanding of climate change and develop activities that promote behaviour change and inspire individual and collective actions that will help the community to adapt, and mitigate, its impact on the local and global environment.  The programme targets areas of Sunderland (Hendon, Millfield, Pallion and Southwick) which suffer some of the highest levels of fuel costs in relation to income.  The programme is led by the Sunderland Black and Minority Ethnic Network and is delivered by a partnership comprising of eight further agencies which include:

  • Age UK Sunderland
  • Gentoo
  • Groundwork North East
  • International Community Organisation of Sunderland (ICOS)
  • KNW Partnership
  • The Council of the City of Sunderland
  • Sustainable Enterprise Strategies
  • Voluntary & Community Action Sunderland

Funded by the BIG lottery in April 2013, Sustainable Sunderland commissioned the Institute of Health and Wellbeing at Leeds Beckett University to undertake the evaluation of the programme.

 

This report synthesises three years of evaluation activity drawing together both perspectives from those intimately involved in the strategic direction and delivery of Sustainable Sunderland and those individuals and communities benefitting from the programme activities.  The report is divided into four discreet sections.  Section 1 reports on early qualitative data gathered through two workshops with the programme steering group.  The aim of this section was to develop and refine Sustainable Sunderland’s Theory of Change.  Section 2 explores community perspectives on the Sustainable Sunderland initiative – this data was gathered at the 24 month point of the programme.  Section 3 revisits the views of the programme steering group toward the end of the funding period, focussing particularly on achievements, successes and challenges of the programme.

 

How did we gather the evidence?

This report presents three sections which draws on a range of evidence and data sources.  The main body of evidence is drawn from qualitative work, exploring perspectives of both those involved in strategic and operational matters of Sustainable Sunderland and those communities where activities and interventions have been focussed.  Table 1 summarises the data sources, the numbers of people engaged in the process and analytical strategies.

 

Table 1.  An overview of data gathering and analysis

 

Aim of data gathering Approach by which data were obtained Number of participants Analytical strategy
To develop Sustainable Sunderland’s Theory of Change Two workshops with strategic partners plus analysis of existing documentation. Workshop 1: Ten partners and stakeholders (representing 6 partner agencies)

 

Workshop 2: Fourteen partners and stakeholders (representing ten partner agencies)

Facilitated by a member of the evaluation team, a discussion and preliminary ‘map’ was designed of the preconditions required to bring about the long-term goal of the Sustainable Sunderland Programme.  Workshop discussions were analysed thematically and emergent themes used to inform the ‘map’.
To explore community perspectives on the Sustainable Sunderland programme In Autumn 2014, all Sustainable Sunderland partners were invited to recruit community members to participate in a focus group discussion with two members of the evaluation team.  For a myriad of reasons, not all delivery partners were able to support this activity; however, four partners were able to recruit individuals that had engaged with their services. Four focus groups were therefore conducted with 22 community residents living in Hendon, Millfield, Pallion and Southwick. Discussions were audio recorded and later transcribed.  These transcripts were analysed thematically, following standard guidance in the literature and cross-cutting themes across the data were identified.
To revisit the views of the programme steering group toward the end of the funding period, focussing particularly on achievements, successes and challenges of the programme.

 

All steering group members were contacted and invited to discuss their views and perspectives on the programme. Telephone interviews with seven strategic partners.

 

 

  

Section 1 – Understanding the programme theory

 

The first phase of the evaluation was to develop an understanding of the programme’s Theory of Change as it was anticipated that this would provide a framework for subsequent evaluation activity (Judge and Bauld, 2001).  The Theory of Change approach is a way of modelling how change will happen in a programme or intervention and is ideal to conduct at the start of a programme or intervention.

 

Theory of Change is an approach, not necessarily a prescribed methodology (Green and Tones, 2010).  Since its inception, Theory of Change has been a popular approach for evaluating complex social programmes which often involve interventions with multiple components leading to multi-level outcomes. The Theory of Change provides a means of unpicking the steps along the pathways of change – or indeed the complex networks. It involves ‘surfacing’ the latent theory which outlines stakeholders’ expectations about the various steps along the pathway linking activities to the achievement of goals. This is done through a facilitated process which draws on existing knowledge and theory and also the insight of practitioners and other stakeholders. Evaluators and stakeholders work together to ‘co-construct’ the theory of change for an initiative (Green and Tones, 2010).  The process of creating and critiquing a Theory of Change encourages delivery partners to be explicit about how resources will be used to bring about the preconditions of the long-term programme goals.

 

The advantage of using a Theory of Change approach is that it helps partners and stakeholders make explicit the links between activities delivered and programme goals.  These programme goals are:

  1. An increased awareness and understanding of climate change and temperature variations amongst residents, community groups and businesses leads to actions being developed that promote behaviour change to minimise the impact of climate change on people in Hendon, Millfield, Pallion and Southwick.
  2. There is a reduction in the impact of climate change and temperature variations on residents’ lives, and their fuel and heating bills, because of changed behaviour and the increased use of energy efficiency measures.
  3. People in the local area make use of the opportunities for new businesses and jobs that arise from the impact of climate change and temperature variations.
  4. The increased understanding of the idea of ‘One Planet Living’ in Hendon, Millfield, Pallion and Southfield supports changes in behaviour that help residents, businesses and community organisations to save money and mitigate the impact of climate change and temperature variations.

 

Ten partners and stakeholders (representing 6 of the delivery partner agencies) were brought together to develop and agree their ‘theories of change’ – this was a process of co-construction between evaluator and practitioner (Green and South, 2006).  The following stages to develop the Theory of Change were:

  1. a) Identification of long-term goals and the assumptions behind them.
  2. b) Backwards mapping to connect to the preconditions or requirements needed to achieve the goal.
  3. c) Identification of the actions undertaken to achieve the desired change.
  4. d) Developing indicators to measure outcomes to assess the performance of the initiative.
  5. e) Writing a narrative explaining the logic of the initiative.

 

Facilitated by a member of the evaluation team, a discussion and preliminary ‘map’ was designed of the preconditions required to bring about the long-term goal of the Sustainable Sunderland Programme.

 

Key findings informing the Theory of Change

 

The following themes presented are those that emerged through the Theory of Change discussion workshops which took place in December 2013 and April 2014.  Where quotations have been used to illustrate issues, these have been anonymised to protect the participants.

 

Early challenges

When the programme launched in April 2013, delivery partners suggested that there had been early successes and moreover some initial struggles.  One of the primary achievements was the collection of diverse, but complementary, partners that had agreed to be involved in the Sustainable Sunderland programme.  Many of these partners had long and established reputations and, to some extent, were already embedded in the fabric of Sunderland.  Each partner was reported to bring their own unique expertise and ‘client-base’ (e.g. Age UK, Sunderland Black and Minority Ethnic Network and ICOS):

“Each of us brings our own trusted constituency of people we work with.”

Further accomplishments were reported in relation to early programme delivery being undertaken and this work contributing to proxy-outcome measures.

 

There were, however, some early challenges associated with the Sustainable Sunderland programme.  First, there was a concession that the partnership had struggled to start, with partnership agreements taking time to resolve.  One partner, for instance, noted:

“It’s taken a while to find our way as a partnership.”

Moreover while partners commented that the programme had a suitable name, they did not feel that they had a clear identity as yet.

 

A developing partnership

The partnership is diverse and covers nine agencies from across various sectors; however there were initial concerns raised that the partnership had not yet been truly formed.  Rather it was perceived that the programme comprised a collection of nine partners instead of an integrated and coherent partnership.  Explanatory frameworks were proposed as to why full integration had not yet occurred; the relative infancy of the programme was touted as one reason, but for others it was a case of time being required to break-down boundaries and understanding partners’ ways of working.  This was neatly encapsulated by one partner:

“We all came at this from a different base, we’ve got some large organisations, small organisations…it’s going to take a while to break barriers down, you know, how do we work with you?”

There was general agreement that partners were working in ‘silos’ and not integrating as a coherent body or harnessing the potential and synergies between organisations.

 

In the follow-up workshop in May 2014, trust between delivery partners was perceived to have increased both at strategic and operational levels.  These developing levels of trust may be responsible for the increased collaborations between partners and some examples of cross-referrals.  This was illustrated by one partner’s comment:

“I know having spoken to our delivery staff that we are working much more closely with other organisations, including some cross-referrals…It’s not happening with all partners, but things are moving forward.”

Synergies were reported to have occurred where there were ‘natural’ affinities between partners’ organisational objectives or styles of working.  Some partners were less clear on how they would work with specific agencies and conceded that more time was needed to explore potential possibilities.  Indeed, it was still apparent that not all partners were cognisant with each other’s contribution to the Sustainable Sunderland programme:

“I’m not sure we still fully understand what each of us does.”

 

One partner suggested that the notion of a true partnership amongst all delivery partners was perhaps unrealistic and unobtainable because of the nature of how the partnership had been formed in light of the BIG Lottery funding:

“You can’t have a true partnership, but perhaps clusters of organisations working together.”

 

Marketing and reach

“It’s not quite sold out there yet….”

In the early part of the programme, marketing the Sustainable Sunderland programme was largely acknowledged to be both complex and challenging.  Partners seemed to agree that the marketing of the programme was not yet suitable and that further developmental work was needed in order for the strategy and execution of the promotion to be correct.  At present, partners suggested that the message emanating from Sustainable Sunderland was at best mixed and sometimes blurred.  A universal ‘strap-line’ was, to date, missing and partners had mixed feelings on what that all-encompassing message should be – saving money; saving the planet; and saving lives through excess winter deaths, were all put forward as reasonable suggestions.  It was clear that the overall strategic approach to marketing had not yet evolved.  Partners were still unclear on the overarching message that should be communicated across the four target communities and this was regarded as a programme weakness:

“I think we suffer from a lack of uniformity of message…I think we’ve ‘missed a trick’ in not formalising something.”

The complexity of the challenge was reported to be amplified in the specific areas in which Sustainable Sunderland were operating (i.e. Hendon, Millfield, Pallion and Southwick) where ‘selling’ the programme’s message and benefits may be viewed with potential scepticism.  Nevertheless, it was argued that the marketing of the programme would become progressively easier as referrals were made and success stories were relayed within communities either formally, via tailored marketing, or through ‘word of mouth’.

 

The referral pathway

Assumptions were made that if the marketing of the programme was correct and referrals came in, worthwhile interventions that would make a difference to people and communities lives would occur.  These interventions could be the installation of boilers or loft insulation, for example, or through training and education. Indeed, it seemed that many of the partners’ expertise were in the delivery of interventions (e.g. KNW and installation of boilers, Groundwork and the Green Doctors Project or ICOS and providing advice and guidance on ‘One Planet Living’).  Currently, however, partners were unclear of the chain of events or referral pathway leading to individuals and communities making environmental changes to their lives.  The analogy of playing in a football team and not knowing who to pass to was useful in encapsulating where the programme implementation was at.

 

Establishing and working toward intermediate outcomes

In order for the Sustainable Sunderland programme to meet its stated outcomes [1] in 2016, it was accepted that having intermediate outcomes which would contribute to the achievement of the final outcomes would be useful.  This not only develops programme logic but offers a practical way of moving forward, recognising the contribution that partners are making.  It was suggested that there were a number of intermediate outcomes:

  1. Raising awareness of climate change in Hendon, Millfield, Pallion and Southwick.
  2. Increasing knowledge around climate change with individuals and/or businesses in Hendon, Millfield, Pallion and Southwick.
  3. Changing attitudes about climate change with individuals and/or businesses in Hendon, Millfield, Pallion and Southwick.
  4. Modifying the behaviours and practices of individuals and businesses in Hendon, Millfield, Pallion and Southwick.

Delivery partners were asked to comment on their contribution to each of these intermediate outcomes and, understandably, some partners’ contribution were more focussed on certain outcomes than others given their organisation’s raison d’être.  However, it was suggested that some of the intermediate outcomes were easier to meet than others.  As an example, partners were more able to articulate how they were able to raise awareness with the target communities, but less clear on how attitudes or behaviours would be modified.

The difficulty in clearly articulating how partners would meet some of the intermediate outcomes led to some scepticism and reservation over whether the programme outcomes for the programme would be fully achieved within the timeframe.   

 

Sustainable Sunderland’s Theory of Change

The following was a logic model, or theory of change, that was inferred from the workshop discussion.  The Theory of Change suggests the preconditions necessary to achieve the programme aims for 2016.

 


 

 

 

 

 

Section 2 – Community perspectives on Sustainable Sunderland

 

The focus of this section is to report community perspectives on the Sustainable Sunderland programme.  Data here was derived from individuals who had been involved in the activities and interventions delivered by the programme.  Data were gathered at the 12-18 month point of the programme.

 

The activities and intervention

 

All participants had received an intervention delivered by a Sustainable Sunderland delivery partner. These included: advice; information on energy saving in the home; small modifications to their houses installed by a delivery partner; and advocacy with energy companies. One group mentioned being given information about One Planet Living.  Participants talked about delivery partners giving:

  • Advice on how to save energy by, for example, not over-filling the kettle or leaving devices on standby.
  • Information about the different wattage of household items.
  • Information on how to use their heating most efficiently.
  • Installing energy saving light-bulbs, reflective panels behind radiators and draft excluders.
  • Information (or help) to switch energy suppliers.
  • Liaising with energy suppliers in regards to unexpectedly high energy bills (advocacy).

 

The energy saving advice given was appreciated. Whilst many already knew how to save energy it was seen as a good thing to be reminded – particularly as some felt they had got into bad habits around energy-saving:

“He gives you simple things that you probably already know, but I think it’s good to have reminders, like you change your light bulbs, the thing in the toilet. So, you put kind of a block in the toilet, so you only use half the amount of water. And then a drain in your sink. The, the things we already know, but it’s good to be reminded.”  (FG 1, P1)

Individual advice on heating systems was also very much appreciated:

“He showed us about the, it’s like I say, my controls on my boiler and things like that and my settings and how to put timers on and things like that, because I didn’t have a clue how to go on with them at all, So, and my bills have come down.” (FG1, P4)

Modifications to houses were welcomed. Reflective radiator panels helped keep the house warmer:

“the panels have done a, quite a difference, you know, there’s, the solar reflect panels are great.” Draft excluders also improved homes: “It’s no wind through the window, and through the doors … the house is better, better warmed” (FG4, P1 – via a translator).

Help switching energy suppliers was appreciated by participants who came from overseas and for whom English may not be their first language. One participant had been paying via a pre-payment meter and described (via a translator) this process as being “horrible for them because it’s cold” and cost prohibitive. The delivery partner had helped this particular individual to switch supplier, drastically reducing their costs and increasing their comfort.  Switching energy suppliers is discussed in more detail in section 2.

Having an inexplicably high bill was very distressing for participants.  Delivery partner’s support in sorting these out was greatly appreciated by the participants affected. One had been told she was £400 in debt to her energy company, with a monthly bill of £108, despite doing everything she could do keep her bills down. She and her family had tried to resolve the bill but failed. The delivery partner took her case on-board, resulting in the debt being cancelled and her monthly bill halved:

“She took my bills away and everything and I got, these people, (names delivery partner) they came along, two ladies, and they said … oh, well, you’re certainly taking precautions against the cold. I said but I can’t get my bill down from being £400 in debt. I said no matter how much, and I said now, I’m £108, and I said I’m afraid when I’m gonna end up having to pay. So she said oh, so they took all my stuff away and they were marvellous, absolutely marvellous….. Ultimately what happened to me. My £400 debt gone and I don’t have to pay anymore now than £64… I was delighted.” (FG2, P1)

 Another delivery partner had acted as an advocate for a woman recently arrived in the UK – she’d been given a large bill, despite not yet moving into the property:

“I had no heat, I had nothing. And it was light nights, and so I came here, and I said I don’t understand this, I don’t even have a kettle, I don’t even have a radio, but they insisted it was my bill and (delivery partner) sorted it out for me. And they said it wasn’t the old lady that had died, it wasn’t her bill, it was my bill. What a tough time. I really appreciate (delivery partner) because they did sort it out.” (FG1, P1)

With the exception of acting as advocates, there was a general feeling that delivery partners had a fairly limited scope in what they could provide:

“he came out, evaluated what our needs were and what they could provide within that scope. .. within the scope of what they could do, they did do it.” (FG3, P1). 

Many felt they were already careful in regards to their energy use and were aware of how to switch suppliers. This partly relates to the inadequate condition of many of their homes – see section 3.

Behaviour change that people cited included: not over-filling the kettle, using a more economic programme for their dishwasher and using more energy efficient light-bulbs. A couple of participants talked about how, having attended a One Planet Living workshop on seed-planting, they now grew their own vegetables.  Positive outcomes cited by some participants include houses now feeling warmer; spending less on heating; and, for those people who had received help sorting their bills, a reduction in stress and anxiety.

Perceptions of how the interventions were delivered were generally very positive.  The individuals concerned were praised for their professionalism, good communication skills (particularly keeping in touch regularly and even after the intervention) and the thoroughness of their visits. Participants trusted the individuals concerned – they saw them as being very informed and emphasised how much time they had spent with them:

“I find him very practical, very pleasant, very available. He said anytime I wanted to phone him, I could. So I didn’t feel like he talked over your head, he talked to you, not at you” (FG1, P1)

One delivery partner did not always receive positive feedback – this related mainly to their role as a landlord, rather than the specific Sustainable Sunderland intervention.

An issue raised by a number of participants was a lack of follow-up in terms of having in-house modifications installed.  They had had an advice visit where the installation of reflective radiator panels and draught excluders had been recommended – but they were still waiting for this to happen:

“But, you know, I bought some stuff myself to try and do the doors ‘cause I’d asked twice and they just said that they were busy. So, there was no follow up, and (delivery partner’s) a nice fella but all he can do is ask. But we’re into winter now and I just find it, like I, I can’t seal the doors properly. I cannot do it myself.” (FG1, P1)

Most participants had found out about the service from the newspaper or existing contacts with delivery partners. Extending the reach of the programme was recommended. It was felt many more people could benefit from the interventions if they were advertised more widely or more resources were committed to ensuring follow ups took place. Participants could not understand why there was not more demand for the advice offered – given that the advice and help were free!

 

Energy Use

The high cost of energy bills was a major concern.  It was this that served as the main motivation for participants to minimise energy use. Comparing suppliers, tariffs or different technologies dominated the conversations. Many could not comprehend how heating a house had become so expensive:

“It’s too much, it’s far too much. I’ve never seen bills like them, the way we’re getting them now” (FG3, P4)

Switching suppliers was something the majority of participants felt able and willing to do:

P1: Well, I mean, I’m always looking. However it is felt that the savings made by switching are now not particularly substantial: “Like he says, it’s a cartel. They, decide amongst themselves, it’s not much different, that much difference between the price”.

As mentioned in section 1, those people who struggle to switch are often those who are new to the system (for example new to the country) or do not speak English as a first language. In addition, it was pointed out that switching requires internet access and computer literacy –something not every participant possessed:

“You can’t change unless you’re on the computer, and we’re computer illiterate, you know, wouldn’t know how to turn one on. The daughter does it for us, we’re completely alien.” (FG1, P5)

It is not just the high price of energy that caused issues but also the confusion and lack of transparency regarding costs.  Participants felt it was too difficult to compare the costs and effectiveness of different heating sources e.g. a gas fire compared to an electric fire or a new type of heating fan. The number of different tariffs added to the confusion:

P1: But then they start talking back at you about kilowatt hours and this, that and the other, so the fire…P4: I haven’t got a clue.P2: Well, exactly. P1: A kilowatt hour for the fire is totally different to a kilowatt hour for a boiler, and this, that, but why is everybody charged different tariffs? We’ll all consumers.” (FG3)

Concerns about receiving a high bill resulted in people being too anxious to use the heating. One participant, who had been presented with a large bill (that turned out to be incorrect), said:

“I’m frightened to put anything on, I said I’ve got everything switched off I had the heating thing switched off.. because I’m afraid of, I’m afraid to use anything because I don’t know what my bill’s gonna end up being” (FG2, P1)

People reminisced about using coal to heat their homes – perhaps because they knew how much a bag cost and how many bags it took to heat their home.  It is clear that participants did not trust the energy companies – many spoke vehemently about being ‘ripped off’ and that the companies were profiteering “I think they just pay lip service to the complaints that’s coming in … and I think they’re all in each other’s pockets”.  Standing charges were particularly disliked as it was felt consumers were paying for receiving nothing.

 

Housing

Having a poorly insulated house was common amongst those who engaged in the discussions. People talked about how the temperature in their homes fluctuated greatly due to a lack of insulation or poor draught proofing. Having condensation or old central heating systems were also key issues:

“And it’s very cold in that place, even with the heat. I turn the heat on and then I turn it off to go to bed and the place gets cold like [clicks fingers].” (FG1, P1)

Most participants were tenants and felt powerless to improve the thermal quality of their homes. There was a general feeling that landlords were not concerned about the cost of heating homes or how comfortable their tenants were. Some participants were clearly frustrated or angry. One woman was very anxious about the temperature fluctuations in her home, feeling it exacerbated her own health condition and made her children ill. She had raised her concerns with the landlord but they had advised her that there was nothing they could do in the short-term:

“I can’t do really nothing, I just pay, pay more and try to keep temperature quite nice. So, I can’t do any more. When I talked with (my landlord), they advised me about turn on all the time, but it’s impossible, it’s too expensive. They told me oh, they are sorry, they know it’s old and that’s why it’s expensive. I have to wait. When I told them OK, I have had dizzy(?) and ill all the time as well. Oh, they are sorry, but 2017.” (FG2, P4)

 “The other thing is, the wife’s got COPD .. so the house has gotta be warm and believe you me, you, you don’t wanna go there. We should’ve stopped in the towers, should’ve stopped where you are. … Bag of rubbish they are (landlord)” (FG2, P3)

There were some exceptions – for example, a man living in a tower block talked about walking around in his shorts as the flat was so warm. A few individuals praised their landlords for the good quality of their home:

“And they’re good, you know, they, when they rent the houses out, they’re all new doors and, you know, the roofs have been done and they’re good, you know, for anything that goes wrong, you just phone them and, and they come fix it. It doesn’t matter what it is, it’s, you know. But, I mean, I mean, there’s adequate radiators in the house and things like that. They’re alright, you know. It’s just the worry of just putting them on, that’s all.” (FG3, P4)

 

Health

Participants were very aware that having a cold house impacted negatively on their health and wellbeing. People tried to keep warm by remaining in one room, staying in bed or covering themselves in bedding:

I lie on the settee of a night-time and rather than put a heater on, I’ll drag the duvet cover off the bed, you know what I mean, there, because, well, I’m on a zero hours working contract, so up to now, I’ve had no work this month and you cannot go back to the dole and say well… I haven’t had any work, (FG3, P3)

 I had to move because of the heating factor and the bills I was getting. That’s why I moved into a little cottage. And so it’s better to maintain that, I can, you know, ‘cause I’m only, I only use one, sit in one room, I’m by myself, and I just try to keep that one room really basically warm. I’ve turned radiators off everywhere else. I’m just, you’re frightened…You’re frightened to use it. And I sit, and I’m, I’m, I suffer very badly with different ailments and that, like, but circulation’s terrible and I get cold, and I sit and I’m blue before I put the heating on. You know, you’re scared. (FG3, P4)

Not being able to keep them or their families warm impacted both on their mental and physical health. Perhaps most insidious was the worry and the stress being cold caused, with many participants talking about dreading the arrival of winter:

“It’s killing me actually because I have heart disease and this temperature it’s, I’m, I’m cold all the time. Stress, you know, I’m stressed all the time as well because I have to put money, much much money to my gas meter and I see my children are ill, I’m cold all the time, so I have to take more medicines and after a few months my doctor gives me more and more and more because I feel pain much, lots of pain, and no, it’s like circle. I can’t do nothing and it’s worse and worse and worse.” (FG2, P4)

“P1: It affects your mood. Cold affects your mood. P5: Of course it does, certainly. P4: I’ve gotta say, you don’t wanna get out of bed do you? I’m freezing. P5: I mean, when you get out of bed in the morning, you say, oh, nice warm bed and then you’re getting up, you’re freezing” (FG1)

 Participants cited physical health conditions exacerbated by the cold. One had experienced pleurisy and pneumonia, a participant’s wife had COPD, whilst another suffered from heart disease and her children were often home from school ill.  There was a clear sense that not being warm led onto more serious illnesses and the use of health services:

“I think when you’re cold, it affects you. You can get sick, then you go into hospital and you’re hospitalised, whether it’s pneumonia you had or and all of that in the bigger scale, that is taxing on the system. So, it might only be somebody saying “oh well, it’s cold on a morning, I can’t get out of bed” but all of that, I call is stockpiling, one thing leads to another. … There’s all, the basic things in life, I always find, if they’re not covered it can branch to all kinds of things.” (FG1, P1)

Emerging from these discussions was a sense that being comfortable and warm at home was something everyone should be able to expect. A real sense of frustration was evident. Many emphasised that they worked (or had done) and paid their rent in full regularly – yet, they still were not able to keep warm without worrying about bills.

“I mean pulling the duvet up, aye, that, that’s one thing. Should you have to do that? You should feel comfortable in your house, with a comfortable sort of pyjamas on or whatever. But, and I think, I mean, we are not well off, but we’re not poorly off, but at the same time, even we worry about the heating, how long it’s been on.” (FG1, P1)

 

Branding

The Sustainable Sunderland programme as a whole is not something participants were aware of or associated with. Their relationship was with the individual delivery partners and the interventions they had received. One group was aware of One Planet Living as the delivery partner they had engaged with had run workshops and information sessions themed under that name.

When the name Sustainable Sunderland was mentioned there was a tendency to link it with the council as a whole. This is a potential issue as many participants lacked trust in the council and individual councillors.  Participants felt they were not interested in helping local people, things did not get done and there was a lack of transparency in regards to council business.  Many participants were clearly angry and verbalised this strongly. The City of Sunderland was felt to have gone ‘down-hill’ – with litter and boarded up shops cited as evidence for this – it was compared unfavourably to other nearby cities.

 Some participants felt local community organisations were in a better position to manage any activity to prevent climate change. They were trusted more, they’re “honest and trustworthy and want to help the community” (FG3, P1) and, as they were local, they could adapt activities – it was emphasised that what works may also vary from estate to estate and that a one-size-fits-all model may not be appropriate. Participants emphasised it was important that any programme to manage climate change was fully accountable and less formal than current council structures:

“…What we’re saying about Sunderland council. If, if Sunderland were to run this sort of thing, it wouldn’t work” (FG3, P3)

“I think a (..) group built around the community, within the community, used by the community, run by the community and (participant) used the right word there, accountability and honesty and openness. You know, (…) the council is getting more and more unaccountable” (FG3, P1)

 

One Planet Living

Levels of belief and concern about climate change varied. The majority of participants believed climate change was happening –strange weather and the ice caps melting were cited as evidence.

“Very worried about it (climate change), absolutely. When you see these, you know, up in the North Pole and South Pole, that ice is melting. It’s definitely melting, you can see it.” (FG2, P1)

 “Well, I think we, we all are thinking about it (climate change) because all, we are all reading about it all the time and it’s not on the world weather, is it, because sometimes, you have a nice summer, sometimes you have a bad summer, but in the middle of the years, you can, can see how the temperatures is growing, how really.” (FG4, P2)

 “I mean, sun shining, warm in November, you know, it’s insane, you know. October was absolutely beautiful and you just not normal.” (FG1, P5)

It was felt by some that the impact in Sunderland would be particularly strong:

I agree with that lady there, we all gonna get drowned… it’s gonna come straight up that river” (FG2, P1)

 Concern about climate change appeared to be greatest amongst those who were slightly younger (in their 30s) and those who were older or had grandchildren:

P3: it’ll affect our grandchildren, not us.  P8: Until my grandchildren came along, I start to realise, you know, there’s a future. You’ve gotta think about your grandchildren. (FG1)

Some participants (in one group) were cynical about whether climate change was man made or whether it existed at all:

“how long do they think that climate change is, is it really happening? You know, ‘cause when you go back to the 30s and 40s with all the smoke and the coal fires and stuff like that, now they’ve gone away, Sunderland’s been a smokeless zone for donkeys years so that, and they’re trying to say the climate’s still getting worse. Is it really because of that, or is it really just a natural change, you know?” (FG3, P3)

 Participants lacked a clear understanding of what could be done to tackle climate change at a local level – some were very keen for more to be done about it, others were more fatalistic.  People cited a number of actions they took individually to help combat climate change. The most common by far was recycling their rubbish (the systems to do this were often criticised). Other actions included:

  • Reducing energy use
  • Reducing use of water
  • Not wasting food (food waste was seen as a particular issue)
  • Taking their own plastic bags shopping
  • Cycling rather than driving
  • Holidaying locally

When asked what the council could do to tackle climate change the most frequent suggestions related to buses. Participants wanted greener buses, with more coordinated time-tables.

 

Power and influence

Two more abstract themes ran through the focus groups.

An overriding feeling of powerlessness affected many participants. They felt powerless to keep themselves and their families warm by improving the condition of their house or having the heating on more regularly.  A key aspect of this was their relationship with some larger organisations such as their housing provider, their energy company or the council. It was felt these organisations did not listen, did not take action and in general were not on ‘their side.’ Some participants reacted angrily to this lack of control, others became anxious:

P4: “Everyone here wants their houses warmer and that just doesn’t happen. Not, you, you’re going through the winter freezing, it’s as simple as that, most of us will.” P1: Yes. P4: Nothing is getting done.” (FG1)

 P1 I don’t think the government’s doing enough to address the energy companies themselves because to me, it’s a big word, but they’re profiteering (…) the cost of energy, I mean, it’s coming down what they pay at wholesale and yet, they’re not passing it on. And they’ve been told to pass it on and they’re not doing it” (FG3, P1)

 P4: It’s very frustrating because we are really care about house. We pay our rent every week, never later…. We don’t have any benefits, so it’s, you know, we pay straight away from our account and we haven’t asked about anything else, just please do something about the heating. P2: They won’t, they won’t. P3: I pay full rent right, now all I live on is my army pension, ‘cause I done 22 years in the army, right, and they just rip it off you, council tax, full, full whack. And you’ve got (inaudible), don’t want to go there…. (FG2)

Their experiences with the delivery partners contrasted with what they had experienced with these larger organisations.  It was felt the delivery partners listened, responded and spent time with them – in essence, they were able to build up a personal relationship with them, rather than trying to get through to organisations they felt were unaccountable or uncaring.

A strong feeling of unfairness and inequality permeated many of the focus groups. Whist some participants accepted their situation; others compared theirs unfavourably with other members of society in other geographical areas. It was felt that they should be able to stay warm in their home and be comfortable enough financially to heat their homes. Some emphasised the contributions they had made to society.  This inequality was felt in terms of the North vs. South, Sunderland vs. other North Eastern cities and in general, those with or without influence / adequate resources.

 

 

 

Section 3 – Delivery partners’ reflections on Sustainable Sunderland’s achievements and challenges

 

This section revisits the views of the programme steering group toward the end of the funding period, focussing particularly on achievements, successes and challenges of the programme.

 

Reflections on the partnership

 

There were diverse reflections on the Sustainable Sunderland partnership and the process by which organisations had come together over the 3 year period to reach common goals.  Some individuals were confident that the number of partners and the contributions that each brought was well-balanced and complementary, other individuals felt that a large partnership made communication and practical delivery challenging.  It was clear from discussions that some sub-partnerships between a smaller number of organisations had emerged perhaps because of these challenges – these partnerships were often formed around areas of mutual interest or historical relationships and tended to work very effectively.  It was suggested that there would be a strong likelihood of these sub-partnerships continuing to work together after the contractual arrangements of Sustainable Sunderland had ceased.

A minority of respondents were critical of several partners’ original motives for engaging with Sustainable Sunderland and questioned their background and experience in relation to environmental matters:

“Some of them [partners] had never even uttered the word environment in their life”.

Several respondents suggested that partners’ activities and contributions to the aims of the programme were “at best tenuous” arguing that some partners were ‘shoe-horning’ their activities and making it ‘fit’ the environmental focus of Sustainable Sunderland.  One respondent went further to say that some organisations did not even adapt existing work to the project, they continued with exactly the same activities, but under Sustainable Sunderland:

Some of the other partners were sort of already delivering that work anyway […] a lot of the people round the table weren’t delivering anything over and above what they were already there to do.”

There was not a consensus on whether the right partners had been involved in the programme.  Some individuals suggested that some partners brought very little to Sustainable Sunderland and:

“still to this day do not contribute anything valuable”,

others though argued that an extension of the partnership would have been beneficial to achieving programme aims.  The local council, Sunderland University, Sunderland College and organisations representing private landlords were mentioned.  Another suggestion was that more grassroots organisations should have been involved, with the idea that by including smaller partners, they would have had more reach to the people they have contact with in the community. Several partners thought that giving out small grants to local organisations, as happened in the BIG Lottery funded project in Newcastle, would have been another way of increasing impact and engaging the community.

Overall, respondents suggested limitations with the partnership and a reluctance to be innovative and radical in approach, but also outlined its strength as a diverse body representing a wide demographic.  Moreover, some suggested feeling that as the programme progressed as too did trust and understanding between organisations facilitated by informal conversations and formalised steering and operational group meetings.

 

Geography and context

 

A reoccurring issue to emerge from respondents was that neither the geographical focus of Sustainable Sunderland on the four distinct wards, nor the political and community context facilitated effective working.  In relation to the former, the focus on four areas only was felt to ‘hold back’ the project and limited interventions within the city centre.  The diversity of the wards in terms of community profile and location was seen as being inhibiting rather than enabling.  A city-wide strategy and programme may, it was suggested, have heightened visibility and potentially offer greater returns.

In regard to the wider political and community context, some respondents felt that the general cutbacks in people’s income and standards of living had resulted in ‘belt-tightening’ which meant that focussing on long-term environmental issues and concerns (i.e. climate change, global warming etc.) were of limited concern.  Whilst there is little that Sustainable Sunderland can do in relation to the wider political landscape, it was an issue that individuals felt hindered the communication of some of the messages from the programme.  In addition to this, one respondent felt that some of the language used to convey the key messages of Sustainable Sunderland was too “highbrow” and meant that people were less likely to engage with the project.

 

Governance and management oversight

 

Again, there was diversity in how respondents perceived the way the programme had been managed and overseen.  Those describing themselves as ‘smaller’ partners, for example, mentioned how they felt ‘equal’ within the partnership and able to express views openly in steering group meetings.  Given that other partners in the programme were much larger, there was a concern that voices could go unheard from smaller organisations – this was not the case, however.

There were some concerns though that some partners were able to:

“get away without doing very much”

and that the steering group and project manager should have held organisations to greater account.  As an example, it was suggested that partners should have been challenged further on issues in relation to underspend and on missing quarterly or annual monitoring targets.  As one respondent suggested, there needed to be:

“more stick and less carrot”

from those leading the partnership.  Another respondent echoed this sentiment stating that the management could have taken:

“it [Sustainable Sunderland] by the ears, given it a shake and took things further.” 

There was a feeling of frustration from several partners that things were discussed in the meetings, but not followed up.  The management, on the other hand, felt limited in what they could do as there wasn’t leverage to withhold funding and it would have been difficult to gauge how much money to deduct in any case.

The overall strategic direction for the project was sometimes criticised for lacking focus and in addition decision-making processes were sometimes accused of being slow.  Clearer policies and systems were perhaps required to ensure that partners fulfilled the obligations that they committed to at the inception of the programme. It was also thought that defining the roles of the partners clearly at the beginning, regarding what they were specifically going to deliver as part of the programme, would have helped to give Sustainable Sunderland more focus and a stronger identity. A “collegiate” approach to management rather than “dictatorial” approach was described which may have not always been appropriate.

Furthermore, some participants reflected on and were frustrated by the inconsistency in partners’ tenures on the programme which started and finished at different points.  This was felt to be problematic, haphazard and, in hindsight, detrimental.  Some partners felt that a key factor should have been to ensure that all contracts commenced and ended at the same time.

 

Brand awareness

Mixed views were apparent in relation to the overall ‘brand awareness’ of Sustainable Sunderland. Some individuals suggested that the brand was gaining momentum and was seen as an entity in its own right.  Nonetheless others described the mixed messages that communities had of the programme, still attributing interventions and activities with respective partner organisations rather than a collective body.  This, it was felt, had been problematic to providing clear and coherent messages.  As previously mentioned, it was also suggested that Sustainable Sunderland’s focus on only four wards of the city had prevented the brand from expanding further.  Conversely, others had suggested that the brand had moved beyond the ward boundaries which caused the programme problems, as they were not funded to capitalise on opportunities or demands in other geographical locations in the city.

 

Outcomes and achievements

There was little doubt that the programme had achieved a great deal of success and had met the majority of its milestones and agreed outcomes.  Of note was the fact that Sustainable Sunderland had provided a credible environmental programme, filling a gap in provision that simply was not there in city before its inception.  This is an achievement that should not be understated.

The programme had not been “wildly innovative”, but there was good evidence of engagement with individuals and communities.  Indeed, there was a general sense that the programme had been effective at raising broader awareness of environmental matters and concerns.  The monitoring data gathered during the programme’s lifetime supports this, as figures concerning awareness raising with people, communities and businesses was exceeded.  Some partners did, however, feel that a big opportunity had been missed and that Sustainable Sunderland hadn’t fully capitalised on opportunities and possibilities.  One respondent believed there was too much emphasis on energy saving and for environmental change to happen, you need a wider, more hands on approach.

One issue consistently mentioned, was the challenge of facilitating and enabling behaviour change, both evidencing that behaviour change had occurred for people as a result of the programme and the difficulty in shifting entrenched practices in the timeframe that Sustainable Sunderland had as a funded programme.  A further challenge had been the recruitment of volunteers into the programme which some partners had found difficult. One partner cited the lack of clarity on the definition of “volunteer led work” as a barrier to meeting that target; some partners were recruiting volunteers but they weren’t leading work and in some cases they were only available for a short period of time e.g. students.

Another area where the project struggled to take hold was in offering environmental awareness assessments to businesses.  One reason suggested for this was that bigger businesses were already providing this training, and smaller businesses either couldn’t spare employees for the training sessions or “didn’t want you nosing around in their business.”

 

Future legacy

There was no definitive position on the future of Sustainable Sunderland and the legacy after the BIG Lottery funding.  There was a general sense that if Sustainable Sunderland was to have a longer-term future beyond the existing funding period then maintaining momentum would be critical to achieving this.  Any longer-term gaps in continuing programme activities and interventions it was felt would be detrimental to the continuation of the partnership and in maintaining the impact and outcomes that had already been achieved.  Some were very eager to see the partnership remain intact, while others suggested that partners may “take flight when the money runs out”.  In general, those interviewed believed Sustainable Sunderland would continue by joining forces with other existing groups in the city that have an environmental focus. A hub and spokes model has been proposed by some partners, meaning Sustainable Sunderland will have a looser structure but act as the central hub coordinating several environmental projects across the city.

 

  

Conclusion

 

The evaluation sought to achieve two key aims.  First, to establish the impact of Sustainable Sunderland on the individuals and communities in the wards in which the programme focussed and second, to report on process issues and learning to both inform on-going programme development and to support similar initiatives wishing to replicate the programme.

In relation to outcomes for individuals, our data suggests that there is a clear need for the Sustainable Sunderland programme for residents living in Hendon, Millfield, Pallion and Southwick.  Several of the overriding themes reported from discussions with the community indicated the daily problems faced in meeting the costs of energy prices and moreover, the impact this had on individuals’ ability to keep their homes warm and themselves and their family healthy.

Respondents largely appreciated the work that Sustainable Sunderland’s delivery partners were doing in trying to support them – the provision of energy advice; education on ‘One Planet Living’; and the installation of energy-efficient modifications, such as reflective radiator panels, were all cited as tangible examples.  In several instances, contact with the Sustainable Sunderland programme had been a catalyst for individuals to modify their own behaviour (for example, using eco-settings on dishwashers and changing energy suppliers).  Although the data gathered as part of this element of the evaluation is modest, it does suggest that behaviour change is occurring albeit on a relatively small scale and with, potentially, minimal impact.  Furthermore, individuals’ rationale for behaviour change is not for mitigating the effects of climate change per se, but usually for personal financial drivers in respect to saving money on utility costs.

The awareness and understanding of the Sustainable Sunderland ‘brand’ within communities is problematic.  In short, very few respondents had heard of the programme or were aware of its aims and remit.  Instead, the respondents in communities were far more cognisant of the respective delivery partner – a finding echoed by many of those professionals on the steering group.  Our analysis suggests that the Sustainable Sunderland programme perhaps needed to do more within communities to establish its vision and purpose.

As a forming partnership, there is little doubt that challenges emanated at various stages throughout the programme’s tenure.  However, this should not be surprising as establishing a coherent partnership with nine partners, with sometimes diverse philosophies and aims, is challenging.  The data showed clear instances where tensions between partners had emerged, but often issues were reconciled in transparent ways.  Smaller organisations within the Sustainable Sunderland partnership seemed to have ‘gained the most’ from the collaboration, citing the future potential to work with various partners as a result of developing relationships through the Sustainable Sunderland programme.

There are lots of positive developments from the Sustainable Sunderland programme.  Our data – albeit based on a self-selecting, modest sample – demonstrates tangible instances whereby individuals and communities have modified their behaviour in respect to energy use and climate change.  What happens to Sustainable Sunderland and the legacy moving forward is unclear.  For some partners, there is a clear appetite for environmentally focussed programmes that builds on the work by Sustainable Sunderland – if this is to happen, it seems that many partners would be in favour of a city-wide approach.

 

Learning points

In terms of key learning points that have emerged from the Sustainable Sunderland programme, the evaluation findings would suggest a number of points to consider for similar programmes in the future:

  • Clarity – clearly defined roles are essential for all partners to enable demarcation of role, expectations of service delivery and consequences of not delivering as agreed.
  • Brand – the marketing and PR of a programme which is bringing together wider and diverse partners is essential to demonstrate coherency and understanding for communities. In this evaluation, respondents in communities were far more cognisant of the respective delivery partner than Sustainable Sunderland as a programme.
  • Language and communication – linked, in many ways, to branding, ensuring that the language used and communication strategy to promote the work of Sustainable Sunderland and similar environmental campaigns is essential. Offering both the short and longer-term benefits of behaviour change, both for individual gain and for wider societal gain is essential.
  • Monitoring and evaluation – capturing outcomes and monitoring progress is an essential aspect of any programme. Being clear on what data is being gathered and how it is being analysed is critical.  Moreover, evaluating longitudinal change is challenging but research designs should try to incorporate this where funding is available.

 

 

References

 

Green, J. & South, J. (2006) Evaluation, Maidenhead, Open University Press.

Green, J. & Tones, K. (2010) Health promotion.  Planning and strategies, London, Sage.

Judge, K. & Bauld, L. (2001) Strong theory, flexible methods: evaluating complex community-based initiatives. Critical Public Health, 11, 19-38.

[1]

  • An increased awareness and understanding of climate change and temperature variations amongst residents, community groups and businesses leads to actions being developed that promote behaviour change to minimise the impact of climate change on people in Hendon, Millfield, Pallion and Southwick.
  • There is a reduction in the impact of climate change and temperature variations on residents’ lives, and their fuel and heating bills, because of changed behaviour and the increased use of energy efficiency measures.
  • People in the local area make use of the opportunities for new businesses and jobs that arise from the impact of climate change and temperature variations.
  • The increased understanding of the idea of ‘One Planet Living’ in Hendon, Millfield, Pallion and Southfield supports changes in behaviour that help residents, businesses and community organisations to save money and mitigate the impact of climate change and temperature variations.