Tag: impact

Categories
Universities

Why most university impact studies are flawed

I enjoyed speaking on a lively panel yesterday about regional development and innovation as part of the UIIN conference, relocated successfully from Budapest to Zoom. Together with Matthew Guest from GuildHE we discussed how to better understand the local role of small and specialist providers.

The work builds on experimental alternatives to traditional economic impact studies. I first explored the idea of institutional heatmaps on a post here in 2018, and then expanded on this at a workshop in South Africa later that year. Over the past 12 months I have been working with GuildHE to ‘map’ the impact of some of their members. In yesterday’s presentation I set out why I think the traditional ‘big number’ approach to measuring economic impact is out of step with what places need from their universities. Below I go further and list why I feel these studies are, mostly, flawed endeavours. (I should add that these are my personal views, not those of GuildHE!).

You don’t have to look far to see economic impact studies. My former employer had a flagship biennial report with a steadily-increasing figure for the impact of UK universities – £21.5 billion to UK gross domestic product at last count – which it has used successfully for lobbying and campaigning. As long as this figure keeps increasing, everybody is happy. Many institutions have their own studies – £650 million of impact here, £400 million impact there – and often with LEP-level or regional disaggregation. Of course, such studies are not limited to higher education. We’re informed that shooting contributes £2 billion to the UK economy and supports the equivalent of 74,000 full-time jobs. Ornamental horticulture and landscaping contributed £24.2 billion to national GDP in 2017.

Why we need change

There are helpful academic papers which deconstruct the methodologies for calculating economic impact, and the common pitfalls. Instead, I want to challenge the preoccupation we seem to have with ‘one big number’ impact studies and what we lose in the process.

There are two shifts taking place which render the traditional impact study less effective:

  1. A single large number fails to capture what is increasingly important. The shift towards universities being ‘for’ a place, rather than simply ‘in’ or ‘from’ a place, means this data needs to be far more nuanced. We need to know specifically who is benefitting, and how, and who is missed out. We need to know the businesses and the communities behind these numbers. As disillusionment grows with traditional methods of measuring economic success – GDP, GVA – and attention on ‘inclusive’ and social development begins to be translated into policy change, economic impact analysis needs to keep up.
    Traditional impact studies simply don’t do justice to the range of university activities. They measure spending, output and employment, but do not capture the full impact of engaging with communities in a marginalised neighbourhood, or working with small businesses to strengthen their supply chains, for example – activities that may have huge impact but make little difference to a £400 million impact figure. (Accounting for social value can help here).
  2. As we grapple with recovery from Covid-19, it is both tone-deaf and ineffective for universities to be shouting about how good they are, whilst also asking for assistance from government. Rather than communicating about the size of their value-added, university messaging needs to focus on solutions and partnerships. Policymakers need a more sophisticated understanding of impact which moves beyond broad figures to specific information on which communities, businesses and industries have benefited from the university, and who stands to benefit from future support.

What else is wrong with traditional impact studies?

I should note that economic impact studies are not all bad. It is helpful to see returns on investment, and to raise awareness that universities have economic clout and should be seen alongside other major industries. But they risk being a blunt instrument, obscuring what is often highly patchy and inconsistent local impact behind impressively large numbers. Economic impact studies need to be married to a rich understanding of local impact – perhaps through something like an institutional heat map combined with a survey of perceptions or social impact assessments.

Four further shortcomings that come to mind:

  • Uniformity. Despite huge variation in local contexts across the UK, and the individual histories and missions of universities, impact studies all end up looking pretty much the same. As with my engagement strategies test, if you line up five university impact studies and remove the university name, can you tell who (or where) they are talking about? The uniformity of approach, and measuring success against numerical benchmarks, means we lose out on what may be needed. By working towards what is measured and counted, impact ends up converging into a standardised set of headline numbers and we lose the local context.
  • Impact. Slightly tongue-in-cheek, I would like to see an impact study of impact studies. Do they lead to positive change? Or boost perceptions of universities? Quite possibly. But next time you are in a taxi to a university, ask the driver about the impact of the university. You’re unlikely to be quoted an economic impact figure of £450 million a year to the LEP’s economy. You’ll probably be told about the business that decided to open a new site near the university, or the impact of students volunteering with communities (and how the university is good business for the taxi company – at least before lockdown). You might argue that economic impact analysis is aimed instead at funders and policymakers. But should it not also reach residents and businesses?
  • Fatigue. Somewhat cynically, does anyone really care whether the economic impact is £600 or £900 million? Beyond a certain point, big number fatigue sets in. Figures between institutions are not always directly comparable, and the process of reaching the figures is not always transparent (or easily replicable).
  • Unintended consequences. We are not at this point, but I can imagine a league table of economic impact rankings. Universities should be well aware of the limitations of league tables, and the uncanny ability of rankings to shape and warp policies away from what is important – both for the institution and for the place.

Above all, my concern is that economic impact analysis can mask inequalities and ‘cold spots’ in university engagement. Of course, heatmapping as an experimental alternative brings its own set of issues. Consistency between institutions, subjective judgements over the importance and intensity of shading, and the complexity of trying to map such a wide range of activity are issues that need to be resolved. But they may also expose quite starkly where a university is not working, and not having an impact – things that are hidden in the ‘one big number’ approach.

(Image credits: original images from Unsplash here and here.)

Categories
Universities

How to write an engagement strategy

This post first appeared on Wonkhe.

Writing an effective strategy for local engagement is difficult.

Read through the following statements, taken from a UK university’s strategic plan, and see if anything looks familiar:

As a vibrant knowledge hub, we have an important role to play both locally and globally. The university is a large employer and economic contributor in the region.

We aim to build a clear and distinctive reputation for excellence through strategic engagement and communication with our regional, national and international communities.

We will help the region address its challenges and opportunities while incorporating its many possibilities in education and research.

Could this be from your university’s strategic plan or external engagement strategy? I actually lied at the start: these statements aren’t from a single university. They’re a jumbled mix of statements from five universities from across the UK nations, and from various mission groups. But if they look familiar this could be a problem.

A Barnum effect?

Speaking at recent events, I’ve asked the audience – mostly of higher education professionals – whether these statements about engagement could be from their strategic plan. Nearly everyone raises their hand.

This reminds me, perhaps a little uncharitably, of the Barnum effect. Emerging from a series of psychological experiments in the 1940s, the effect involves showing participants – for example, a class of students – an individual personality statement based on a handwriting sample or a piece of written work. Statements include:

  • You have a great need for other people to like and admire you.
  • You have a tendency to be critical of yourself.
  • You have a great deal of unused capacity which you have not turned to your advantage.

Because the assessment statements are so vague, people interpret their own meaning, and the statements become personal to them. Nearly all participants believed their assessment to be uncannily accurate.

Engagement expectations

There is a growing expectation for universities to be regionally engaged, and this follows from a recognition of the role universities can play in their area and with local communities. The interest and enthusiasm around the Civic University Commission is evidence of this.

This local role isn’t new. Many universities have had engagement as a core part of their mission since their founding. But in recent years government has focused on the role of cities and regions in devolved policymaking, as well as the institutions, such as universities, that can help steer this.

As regional plans – from City Deals to Local Industrial Strategies – shape the policy environment university planners need to factor the issue of place more highly. Often, however, the result is strangely place-neutral: a series of engagement strategies and university planning documents that are largely indistinguishable from one another.

Place, positioning and partnerships

There are positives to be drawn from my (admittedly unscientific) scan of university engagement strategies. Many recognise how the local and international activities of universities reinforce each other. Universities are bridges for their towns, cities and regions to reach the world, and this offer to connect the local and the global is made loud and clear.

In my work for the British Council on universities and cities working together on internationalisation, I found that effective planning was built on a deep understanding of place, positioning and partnerships. It is long-term, deliberate and part of a wider vision of the future of the local area. Universities often have a clear sense of their positioning, but articulating this in the context of place and partnerships can be tricky.

University strategies are not the place to detail individual activities and actions. But a greater degree of specificity is often needed, and this is likely to mean dropping some of the broad statements to focus on a few areas of institutional strength, perhaps joining up with other local universities.

Rooting strategies in place, positioning and partnerships is a good start. The Civic University Commission’s progress report notes that measuring civic engagement is not widespread (but does highlight Cardiff University’s use of quantitative and qualitative analysis). Further work on this is also important.

A good incentive to write a long-term strategy (image credit)https://unsplash.com/photos/cKjxGyfNdQc
A good incentive to write a long-term strategy (image credit)

So too is taking a longer-term perspective. Many universities proudly promote their anchor presence in their area, but their plans are often on five or ten-year cycles. The city of Amsterdam has a 200-year strategy. Although this is prompted by climate change (as an official said, “we don’t want to just let the water flow and all have to move to Germany”), thinking about what a university strategy spanning one or two centuries may look like is perhaps a useful exercise.

A test when looking at your university engagement strategy: if you remove your institution’s name, and mix it up in a pot with other anonymised strategies, is it clear which strategy is yours?

Main image credit

Categories
Universities

Heatmaps: hotspots, coldspots and the bits in-between

Back in 2017 when I was presenting my work on internationalisation for the British Council at conferences I would ask the audience to picture in their minds a big map of a city they knew and to shade in red the areas where there was most international activity.

For most cities, the deeper shades of red would be in the centre of the city: the central business district, the tourist hotspots, the shopping streets and, often, a university (especially if it bears any resemblance to Hogwarts). There could also be ‘pockets’ of internationalisation in more marginalised areas where universities set up a summer school, ran public events, built student residences or held community engagement activities.

The thinking was that universities could help bring the benefits of internationalisation to these ‘cold spots’. I’ve been thinking about the concept of a heatmap of university-city interaction in more detail and sketch out some initial thoughts below.

What is hot?

Beyond international activity, there are many other interesting dimensions a heatmap could capture. A basic map may capture any initiative between universities and city hall, between universities and businesses, or between universities and communities or community organisations. Darker shading may represent scale of activity or depth of engagement or a longer history of working together.

Less tangibly, it could represent informal collaboration, or any activity where the university reinforces the goals of city hall or supports communities, or vice versa. Activity that undermines other actors might emit a chilling shade of blue; a warming red means partners working towards similar goals.

Heat could represent individuals participating in higher education or people otherwise engaging with a university – from attending a public lecture to using sports facilities. It could capture the flow of these people to and from their home or workplace and the university, showing how their engagement is shaping transport use and public spaces. Movement patterns will differ from university to university and each tell a unique story (Toronto’s universities are jointly studying the travel behaviour of 600,000 students).

Instead of mobility, the flow of money or investment in and out of universities could be measured. In doing so we would veer into the territory of university impact studies and input-output analysis. Given the limitations of such studies, a heatmap approach with added contextual data may offer a more complete picture of regional impact. A broader impact heatmap may look at perception data or a combination of economic, social and cultural measures.

A map could show ownership. Most obviously this could be the land and buildings owned by the universities (perhaps a more granular version of this data from the UK showing the dominance of the Oxford and Cambridge estates). In cities that have a degree of ‘ownership’ of institutions (through regulatory controls or funding mechanisms) the degree of autonomy could be mapped.

Or we could (try to) map where collaboration or engagement is less or more than expected. This mirrors nuanced higher education participation data produced in the UK (my post on that here) which maps the proportion of young people participating in higher education compared to that expected given GCSE-level attainment and ethnic profile. How we measure or define what is expected given the different make up of cities and universities is an interesting question, and leads us nicely to…

Mapping complexity

Heatmaps offer a nice visual representation of the heterogeneity and complexity of both universities and cities. ‘The city’ is made up of countless constituent parts, and it is similarly difficult to generalise ‘the university’ as a single actor. Even the most outward-looking university will have departments and teams with strong engagement with people outside the institution and others which remain mostly insulated from outside.

We can apply heatmapping to universities. Here’s the organogram for Hungary’s University of Physical Education (ranked highly in Google Image Search), with a completely fictitious heatmap applied that could apply to international or community or business engagement. You can get even more detailed: within each unit you could shade each individual. And university structures change over time, and in turn so does the heatmap shading. You could do a similar exercise across a map of the campus.

The organogram for Hungary’s University of Physical Education
The organogram for Hungary’s University of Physical Education

A unique heatmap signature

Every city and every university will have a unique heatmap ‘signature’. This is partly affected by the structure of the city itself: a heatmap for Paris would look very different to London or Dublin or Baltimore or Toronto. A long history of city planning, the decisions of millions of individuals and thousands of businesses and organisations, political and cultural and social and economic forces lends urban areas a unique fingerprint. In Paris social housing is concentrated in the banlieues or suburbs that form a ring around the centre of the city, whereas in London social housing is woven into the fabric of the city. The result can be intense spots next to each other, or softer scattered blobs.

Universities are actors that make decisions but simultaneously are themselves shaped by wider forces. Dublin City University is in a historically poorer part of the city whereas Trinity College Dublin is right in the centre, forging their own unique heatmap signatures. In Toronto the four main universities have very different footprints and very different heatmap signatures. In London, three universities that may be seen by some as institutionally similar are engaging in vast campus expansions in new areas of the city. The heat signatures for UCL, Imperial and Kings College London will show a new, emerging concentration of heat in their new campuses, a second centre of gravity which – depending on what you are measuring and the success of their developments – may over time have implications for their existing sites, the surrounding areas and all the bits in-between. London South Bank University is focusing on working with local partners such as further education colleges in the borough of Southwark; again, the signature for LSBU would look quite different to UCL. Precisely where you are located matters.

Heatmaps may also be a good way of visualising activity on the ‘periphery’ – a focus of recent academic inquiry from higher education to smart cities.

A US university's campus map...
A US university’s campus map…
...with a fictional heatmap
…with a fictional heatmap

Is there a dark side to universities?

Not all university impact and engagement is positive. Complaints may be relatively trivial – from students taking over too many houses to making too much noise or not paying enough local taxes. But they can also be more serious criticisms: universities that exacerbate ‘existing cleavages of class and race’ in the race to redevelop and expand their campus, or otherwise reproduce wider inequalities in society. Such conversations often emerge when universities embark on urban regeneration projects – a prime candidate for heat mapping – and the debate often intersects with wider discussions of gentrification and community identity.

The Guardian explored some of these issues earlier this year in coverage of Johns Hopkins University’s ambitious development plans in east Baltimore. The piece quoted several locals:

“This is gentrification, a big institution pushing out a vulnerable community for its benefit,” says Lawrence Brown, a critical urbanist who teaches in the school of community health and policy at Morgan State, Baltimore’s historically black university… Marisela Gomez, a physician and activist in the fight for fair treatment of displaced residents, is blunter. “Every community that’s black and brown and low-income in Baltimore is at risk.”

There’s also an acceptance that the city needs the university. “We need Hopkins to succeed, because that’s the anchor institution in east Baltimore” says the leader of the ‘Baltimoreans United in Leadership Development’ group. And the university recognises the interdependence of the university and the city: “It is inconceivable that Hopkins would remain a pre-eminent institution in a city that continues to suffer decline”.

Baltimore rowhouses (image credit)http://baltophoto.org/rowhouse-5167-BPXN3IC8V
Baltimore rowhouses (image credit)

Needless to say, mapping such interactions needs to be supported by broad contextualisation. And ideally mapping would reflect some other, significant, changes taking place, such as a blurring of the edges around the campus:

With fences, skywalks and forbidding facades broken by loading docks, the medical campus sent hostile signals to its surroundings, and got hostility in return. Assault and theft were common; beggars set up at traffic lights. “Fundamentally it was a hunker-down strategy,” [Ron] Daniels [president of Johns Hopkins University] says. “The traditional thinking was that the best way to protect the university was to ensure that its perimeters were effectively controlled, and that you were creating safe zones within them.” … By contrast, the new office and lab buildings in the EBDI [East Baltimore Development Initiative] feel like they welcome – and want to generate – foot traffic. It is nothing fancy: ground floor retail, some steps and patios, small setbacks creating spaces to meet and gather.

Messiness and other issues

The shift from forbidding facades to open spaces could be tricky to map, as could other ‘novel’ sites such as mixed-use buildings that combine shops, cinemas and lecture theatres, or – to give an example from northern England – five-a-side football pitches on prime university estate to get young people comfortable being on campus.

There are other limitations. Maps can be stubbornly one-dimensional: they often show a fixed point in time, whereas patterns will change from day to night and times of the year. Unless they can show effectiveness or durability or inclusivity there is a risk of giving the illusion of successful engagement; some projects could create bold heat maps despite having largely negative effects.

With the development of ‘smart cities’ you can, in real time, transpose data onto the map. Although sensor information may show supposed engagement, the data is technical and the metrics unlikely to accurately reflect social realities. Maps need to capture phenomena such as ‘splintering urbanism’, whereby urban infrastructure can drive social and spatial inequality.

Lastly, consideration should be given to how to represent regional, national and international dimensions. To pick just one facet of international links, universities that are close to global flight hubs perform better in league tables, and cheap flights mean more research partnerships; similarly places with a direct flight to Silicon Valley raise more venture capital. But these links won’t benefit all people in the city or parts of the university and a heatmap could help us consider how benefits can be spread further.

If you want yet more reading, here’s a short article recently published by NCEE where I set out the long-term focus universities can bring to planning. And I explored this topic in more detail in this provocation on university impact studies.

Main image adapted from photo on Unsplash