“Our experience with these pilot projects has been very positive, with volunteers enjoying the work and some extremely useful datasets successfully gathered from our maps”: Interview with Christopher Fleet, Katie Haffie and Jenny Parkerson

In March 2022, we spoke with Christopher Fleet and Katie Haffie, of the National Library of Scotland, about a three-pronged, crowdsourced maps transcription project that had just gotten off the ground. Katie has since been succeeded as Community Data Harvester by Jenny Parkerson. Today, we revisit that initial interview and touch base with Christopher, Katie, and Jenny about their experiences with this project, which is nearing completion, and what they have learned in the process.

(Archivoz) When we spoke back in March, one prong (the Roy Military Survey) of this project had just gotten underway and another (the Scotways Historic Footpaths Project) was about to begin. Now, all three prongs are active, perhaps some even nearing completion. How close are you to wrapping these up? Overall, have things gone according to plan?

(Christopher Fleet, Katie Haffie and Jenny Parkerson) The three projects have all achieved very impressive results. We now have a fully searchable gazetteer of all 33,523 placenames on the Roy Military Survey Map of Scotland (1747-55). Our second project, launched in April, has traced nearly all of the footpaths from the Ordnance Survey six-inch to the mile 2nd edition maps of Scotland (1888-1913). By the middle of July 2022, over 70,000 footpaths, spanning over 25,000 miles had been traced. In May we launched our third project to gather all of the text from Ordnance Survey 25 inch to the mile 2nd edition maps of Edinburgh (1892-94). All 30,043 transcriptions were recorded in the first two weeks. As well as creating a very detailed gazetteer of Edinburgh from a century ago, the transcriptions will also be useful as a training dataset for machine-learning, hopefully encouraging the more widespread automatic ways of harvesting names from OS 25 inch maps.

As the three projects had quite different aims, we knew they would take variable amounts of time. The Roy Military Survey project kicked off in February, and the final stage of the transcriptions finally completed in early July. We are currently reviewing and correcting the dataset, with the hope that it can be published and made properly available in the next month or two. The Footpaths project began in April, and whilst its main initial aim – to trace footpaths with an ‘F.P.’ abbreviation by them – was completed within the first three weeks, we realised that a larger project to record unlabelled footpaths without an ‘F.P.’ abbreviation by them, was going to be possible and more useful. By mid-July, around 65% of Scotland has been marked off as being “complete”, and so we hope that at the current pace the project will fully complete within the next couple of months.

For our third project focusing on Edinburgh maps, following the successful crowdsourcing of all the names in the first two weeks, we have subsequently been working with various participants to actively categorise and correct these names, and expanding abbreviations to make them properly searchable. We hope this work, and a possible geotagging phase, will be complete within the next two months too.

We knew it was always going to be difficult to predict quite how long everything would take, and so we didn’t create specific time targets. However, we were keen for the main transcriptions work to complete by the summer and for the projects to generally wrap up before the autumn, so we are very happy with the overall timescales.

(Archivoz) Let’s talk statistics. Approximately how many volunteers signed up to participate across the three projects? Did they tend to sign up for multiple projects, or did they tend to focus on a particular one? Do you have any way of tracking actual participation against the original number of volunteers (and is this even a useful metric)?

(C.F., K.H. & J.P.) 653 people signed up across all three projects together, and within this total, most people focused on just one or two projects. Only 168 people (25%) signed up to all three projects, which made sense as they were quite different in geographic focus and content. The footpaths project was the most popular of all, with 421 (64%) signing up to this, but the Roy Project was close behind with 417 people (63%) signing up. 337 people (51%) signed up to the Edinburgh project.

As expected with crowdsourcing projects like these, we were very pleased to receive several hundred people signing up, as we knew a much smaller proportion would go on to make active contributions. For example, for the Roy Project, only 40% of those who signed up actually gathered transcriptions. Over 180 people worked on the initial Roy name transcriptions, and 170 people worked on the Roy second name transcriptions. But numbers steadily diminished and a couple of months later, nearly all the final phase of the Roy transcriptions work was done by just six people! So the larger initial numbers quickly slimmed down into a relatively small and very dedicated group. For our footpaths project, in the first month we saw 41 active volunteers (10% of those who had initially signed up), but by the third month this had slimmed down to just 12 active volunteers. As expected too, a small proportion of these volunteers have done the vast majority of the work. Six weeks into the project when 40,000 paths had been traced, the top three contributors had each traced more than 5,000 paths each, whilst the top 10 had each traced more than 1,000 paths each. As volunteers completed transcriptions, fewer names were available to transcribe and often what was left were the harder cases. So, what we ended up with as each phase passed was a smaller group of more expert volunteers who helped us to correct and complete things.

(Archivoz) Given the staggered rollout of these three projects, how has each one impacted the manner in which the others were organized/presented? Were you able to adapt to project needs as they presented themselves once each one began? Can you give us an example of a situation in which your approach changed, or new elements were added, midstream?

(C.F., K.H. & J.P.) Yes, we learned a great deal as we went along which impacted on each of the later projects. We held an initial “Launch event” for the projects, and following each one, we successively revised the organisation of these, the content, whether to include a hands-on session or not, and how best to structure and present the information. Following the great success and use of the online forum in the Roy Project, we regarded setting up these forums for the Footpaths and Edinburgh projects as being essential, and these forums were well used and much appreciated too.

The Roy and Footpaths interfaces were developed in NLS, and this made it relatively easy to adapt them as each project went along. This was extremely useful too. Very quickly on both projects, there were requests in the forums for personalisation options, being able to display users’ own transcriptions or their own traced footpaths, and we were able to quickly roll these out.

The Roy Editor interface, showing the specific transcriptions available for editing and their totals (upper left)

Another request was to more clearly display and monitor progress. Within the Roy interface, with each successive phase of name editing, we were able to display how many records there were still to go as a live and ever-decreasing number. For the footpaths interface, we were able to present numbers of traced paths from the outset, but we added additional elements as time went by. Initially, we added a total mileage to the footpath numbers too, but then following a request to show “Completed” areas, we added a grid of 10 x 10 km squares as an overlay so that those coloured blue could be marked off as “done”.

The Footpaths interface, showing the new 10 x 10 km grid squares which could be used to mark these squares off as being complete.

Within the Footpaths project too, once the initial goal to trace “labelled” footpaths with an ‘F.P.’ abbreviation by them was mostly achieved within the first three weeks, we added an ability to categorise footpaths into labelled and unlabelled, and colour code these separately in the interface.

The Footpaths Side-by-side viewer, allowing how traced footpaths could be compared to present-day tracks and the wider landscape.

We also became aware from the forums that our volunteers were keen to see the dataset as it developed, and for the results of the dataset to be queried. We provided regular downloads of the dataset in .CSV and .GeoJSON forms, which allowed them to be checked using desktop GIS software. We also set up interfaces allowing the datasets to be queried. For example, the Roy Gazetteer interface allows the whole dataset to be queried and distributions of particular name elements to be visualised:

The Roy Gazetteer interface, showing how the gathered transcriptions could be searched, and in this case, presenting a distribution map of “mill” name elements.

The Edinburgh Viewer, showing how the transcriptions can be searched and displayed.

For the Edinburgh project, which used Recogito as the interface, the development options were more time-limited, and there were some difficulties users had with correcting transcriptions, and also with grouping and ordering text transcriptions. We therefore took the decision to close the transcriptions phase in Recogito once all the transcriptions had been recorded, and sort these typographic, grouping and ordering details using local scripts rather than by crowdsourcing.

(Archivoz) In our first installment, you formulated several guiding questions you hoped to answer as these projects unfolded, and I’d like to revisit some of those. First, you wondered how important it might be for those involved to connect with each other as part of an online forum or community. In the forums you set up for the different projects, was there conversation between participants, or were they mainly a means of submitting questions to the project administrators? How have these forums affected the outcomes of the different projects?

(C.F., K.H. & J.P.) The forums we set up for each project proved to be well-used and well-received – we came to regard these as essential for future projects. 142 people joined the Roy Forum, and it recorded over 210 posts over a 5 month period. 104 people joined the Footpaths Forum, and it recorded 103 posts over a 4 month period. More importantly, in the feedback we have received so far, the vast majority who joined the forums used them very regularly and found them a useful way of raising queries and for keeping up-to-date.

Most of the dialogue in the forums was between the administrators and the participants, with administrators using the forums for providing advice, updates and progress reports, whilst participants used them for raising queries. There was less dialogue directly between participants, but this did happen – particularly in exchanging advice or interpretations of the guidance. The forums were also a useful way of showing participants that the projects were being actively maintained and organised, setting an appropriately cheerful and encouraging tone to the work, and maintaining good morale.

With a mixed and remote audience, the forums also provided a useful means of assessing levels of knowledge and any difficulties with the interfaces. We used these particularly to scale down some of the scope of the Edinburgh project, as by then we had a better idea of what kinds of activity were likely to result in problems and confusion.

(Archivoz) Asked about benchmarks for evaluating success, you mentioned that, beyond merely quantitative results, you wanted to design a project that the participants could enjoy and would find to be a worthwhile use of their time. Having interacted with volunteers for several months now, what is your sense of that enjoyment factor? Have these projects, put simply, been fun? What sort of feedback have you received from those who have taken part?

(C.F., K.H. & J.P.) These projects have definitely been fun and very much appreciated by those who participated. In the survey questionnaire feedback we have gathered so far, 97% of participants said that they enjoyed the projects and 93% said they would volunteer again for another similar project. Many of the comments we received were constructive, suggesting new projects, or pointing out things that could be improved for future projects. We are also well aware of this from the forums, and from direct communications. The forums created a real sense of shared purpose, and of people enjoying the collective endeavour. Many of the participants have enjoyed getting to know and understand maps better, and have become experts in things like palaeography, 18th century surveying, the representation of footpaths on maps, rights-of-way, and of the history and geography of Scotland and Edinburgh.

We are well aware too that these projects have definitely provided a way of us engaging more directly with users in positive ways. Many of those attracted to these projects said that they were keen to help as they had already enjoyed our online maps as users for many years, and wanted to help the Library. The collective engagement on the forums allowed closer links to develop. Many volunteers contacted us to say they were happy to help with checking or revising the datasets, and we have been able to put these offers into really helpful onward tasks. Some users had skills with running scripts to check data validity and errors, which was really helpful too. Other users who became skilled in palaeography were able to be sent smaller chunks of records that had ambiguous transcriptions. As we steadily moved onto first-name terms with smaller groups of people on the forums, it became very easy to discuss different approaches to completing the work. Many of these users also had constructive suggestions on the project itself, which we were able to take on board.

That said, we do recognise that there has been a high fall-out rate from those who originally expressed an interest in the projects, and the positive feedback very much reflects what we have received – which in itself is from the smaller group who saw the projects through to completion.

(Archivoz) I myself did some work on the Scotways Historic Footpaths Project. I quite enjoyed the work, largely because tracing lines on a map was not the sort of thing I expected from a “transcription” project, and I found it quite intriguing. To me, this suggests the broad potential of this sort of endeavor. Has your work on these projects at all impacted the way you will go about planning future work, even if it does not necessarily fall into an identical category?

(C.F., K.H. & J.P.) We feel in no doubt that there is a big potential in map transcription projects if they are correctly conceived and organised. Some key requirements seem to be:

• ensuring we find a sufficiently large pool of volunteers at the outset to cope with the large slimming of these numbers as time goes by.
• finding tasks that are engaging and interesting. It seems as if simple transcriptions (such as in the Edinburgh project) may appeal to people less than transcribing content that might be “harder”, as well as linking the content to different dates of mapping (such as in the Roy project). That said, many people expressed a great sense of relaxation in tracing footpaths, which didn’t require active “thought”. Maps lend themselves to a variety of transcription activities, and in that the content of scanned maps is not accessible as queryable features, there are many other useful tasks that can be done in future.
• finding tasks that people realise cannot (or will not soon) be easily done by machines. Everyone is aware of the developing potential of AI and the automatic recognition of text and features on maps – but at present, some map transcription work can still be done most effectively by people.
• taking advantage of the aesthetic beauty of old maps, and the multiple interests people have in them, especially the ability to connect with and better understand past landscapes.
• choosing the right time period and geographic area for the group of volunteers. The NLS is based in Edinburgh, with a core collection relating to Scotland, so these projects made sense for our community. We are aware that for transcription projects on other areas, we may well need to collaborate with other institutions to attract an appropriate volunteer community.
• having sufficient staff time and expertise to properly organise and manage the projects. Volunteers need to be reassured they are in safe hands, that queries are dealt with promptly, and any problems the applications are corrected quickly. There was often a huge spike of work required at the outset and then a longer tail of ongoing activity needed. The initial spike was often a challenge to cope with!
• making sure the results of the work either are or will be clearly available for onward free and easy use by the community.
• making sure the volunteers feel that their work has been appreciated and credited.

In conclusion, our experience with these pilot projects has been very positive, with volunteers enjoying the work and some extremely useful datasets successfully gathered from our maps. We are definitely keen to run further transcription projects in future, and hopefully will find ways of resourcing the necessary additional staff work to set up and maintain them.

Images courtesy of the National Library of Scotland