Report from the At-large Workgroup

Abstract / OSI2016 Workgroup Question

The At-Large workgroup was the largest and most diverse in terms of stakeholder representation. At-large delegates observed workgroup conversations during the meeting and contributed to these conversations while letting the workgroup teams answer their questions. At-large delegates met during the conference and convened online both before and after the conference in order to develop some high level takeaways based on what they observed. While workgroups focused on answering a particular big question confronting scholarly publishing, at-large delegates had a wide-angle lens on the evolution of these questions and proceedings.

Introduction

Members of the At-Large workgroup had the distinct pleasure of attending and observing a variety of other workgroup sections. Some at-large delegates participated during the entire Open Scholarship Initiative (OSI) 2016 meeting while others attended only a few sessions. Nevertheless, this allowed us to gain a global picture of the process.

Collectively we noted some themes and observations during OSI2016. The observations we present in this paper reflect a variety of viewpoints—including our individual observations of group process and themes, comments made to at-large delegates during sessions and breaks, and our reflections after the fact.

Observations on the Format and Process of OSI2016

The Open Scholarship Initiative set as its goal the creation of a framework for discussion and collaborative action between stakeholders from diverse perspectives and spheres in the scholarly publishing ecosystem. Workgroups met several times over the course of two days for at least eight hours of face-to-face conversation. Each workgroup was tasked with creating a preliminary presentation (during which feedback from other delegates was offered) and a final presentation; teams were also tasked with the creation of a final paper to be completed within two weeks of the close of the conference.

Facilitation training was offered but in general facilitators were not determined in advance; instead, the process of selecting facilitators or speakers unfolded organically and differed from group to group. The facilitator training provided was helpful even if all workgroups may not have followed these best practices. Even so, spokespersons, collaboration, and consensus emerged throughout each workgroup discussion.

Ground rules were limited; the most important rule, according to the organizing committee, was to be “open and respectful.” Other suggestions, albeit not enforced, included the establishment of workgroup-specific ground rules; limiting side conversations and interruptions; and refraining from tweets or other postings about group process and attributions of statements from individual workgroup members.

Formation of workgroups around specific themes was helpful in bringing together individuals with similar interests, if not necessarily similar viewpoints. The process evolved and played out distinctly in different groups. Many teams referred to the guidelines for facilitators, which prompted activities and proposed outcomes for each workgroup session. Some delegates nevertheless felt that more structure would have been helpful to direct the conversation and enhance productivity; it is unclear whether these delegates had taken or had referred to the facilitation training. The topics ranged in focus and scope, some being more practical and answerable than others. In some groups, opposing philosophical positions—for example on moral issues surrounding open access (OA)—provoked lively debates that generally resulted in reaching some common ground or at least respect for others’ viewpoints. To some delegates, however, this appeared to be less an opportunity to discover common ground or achieve consensus than setting up barriers to further discussion. For example, one at-large delegate noted discord among members of a workgroup regarding how OA is mandated in various countries and geographic regions or across diverse academic disciplines; perspectives of OA among commercial publishers, university presses, libraries, and institutional and government repositories; and commercial versus humanitarian goals. This at-large delegate made a comment during the team’s discussion about these differences and was met with negativity from a member of the workgroup. Nonetheless, most at-large delegates were welcomed as observers in the groups they visited, and were often able to provide perspectives that were useful to group work, by offering comments such as “have you considered this (x) aspect?”

It may have been helpful for members of each workgroup to prepare a literature review relevant to each conversation prior to the meeting. Too little reference was made in some workgroup discussions and presentations to relevant existing work, projects, evidence, or data. Five tutorials were offered to delegates prior to the conference to provide a common background on the issues to be discussed. Some delegates expressed the reaction that there was an overabundance of documentation, very little of which was possible to ingest for those with day-jobs. This was particularly true of email discussions, which clearly overwhelmed many delegates. Common expressions were of the nature of “I just tuned everything out because it was too much.” Laying some groundwork ahead of the meeting, including consideration of existing initiatives and results would help ensure that the teams built on previous work by others. More broadly, OSI should consider how it is positioned and differentiated in the landscape of existing initiatives at national and international levels, and should highlight the unique position of OSI.

At-large members also observed that in many cases, workgroups may have “tabled” topics that important, because the issues were too “big” and therefore not able to be thoughtfully addressed in the forum. Workgroup topics frequently overlapped, but a particular workgroup was unsure whether they should address “x” topic because “x” was probably being covered by another team. It was remarked, for instance, that open data is different than open access; this important element was perhaps not thought through as much as it could have been.

At-large delegates noted that providing feedback on individual workgroups was somewhat thwarted by the individual groups’ formats. Most at-large members visited as many workgroups as possible, thereby only visiting each one for a few hours. This provided a snapshot of topics and process, but did not elicit a meaningful understanding of how any one particular group performed. In addition, the workgroups themselves ranged in topic focus from very general to much more specific, so evaluation was not overly supported. Workgroup discussions and presentations ranged from “big picture” to practical, and from pragmatic to aspirational goals. The delegate-at-large role could have been more specifically formulated. Lack of focus can unintentionally present issues when synthesizing and analyzing experience. This may have yielded discussions that echoed outcomes from other conferences, workshops, or meetings, without resulting in any new findings or recommendations.

At-large delegates noted a considerable, if not surprising, overlap in themes. Some workgroups and individual delegates remarked that a greater awareness of the progress and thrust of other workgroups would have been helpful; some indicated that it seemed as if teams were operating in isolation and delving into ideas and questions already being considered by other workgroups. The preliminary presentations addressed this need but may have come too late for some; it’s difficult to consider other ways this could have been addressed, however, under the time constraints of this conference. In hindsight, summarizing the groups’ work through presentations on the second day of the meeting proved sufficient, thus final presentations on the third day were perhaps not as fruitful; some delegates expressed the opinion that the Friday morning session was overkill. This final session time, when less energy was available and some delegates had started to leave, could have been more productively used as an open forum and extended discussion on moving forward.

Changing Scholarly Outputs and Changing Needs

Among the most common themes of the conference, discussed throughout most of the workgroup sessions that at-large delegates observed, were the changing nature of scholarly “publishing” products and how these in turn both reflect and affect the changing needs of the academy. Changing scholarly outputs include products of the digital humanities and other digital publication; the emergence of non-linear narrative formats and new forms of storytelling; data publication and visualization; and other research outputs. Comments and discussions centered on changes to the peer review process; the need for change in the promotion and tenure process at most universities; a need for more descriptive and inclusive credits beyond authors and acknowledgements; how this affects decisions on how to curate, cite, preserve, and normalize data; and how the impact of these varying forms of scholarship is measured.

In many workgroups, these issues of “nontraditional” scholarship, digital publications, and digital humanities products were considered significant but somewhat off the main thrust of the workgroup’s agenda, and thus, constrained by time and the need to create a cohesive presentation in two days, these issues were often placed in the “parking lot.” In other groups, such as “What is Publishing (1)” and “Access and Preservation,” these issues influenced the thrust of the presentation even if, ultimately and not surprisingly, “answers” to these issues were not provided.

From the perspective of scholarly publishers, librarians, and other stakeholders represented at OSI, there is an eagerness to bring about an evolution of peer review as it develops in the 21st century. Questions surrounding peer review emerged at some point in the process during most team discussions, not only in the “Peer Review” workgroup. Questions raised frequently included whether peer reviews should be open, whether peer reviewer names should be made available, the role of post-publication peer review, peer review of data, and so on. There is agreement that the perceived and possibly ageist, sexist, and racist biases that may have pervaded the peer review process for hundreds of years are finally being addressed. Publishers need to be prepared to accept new methods and assist in putting them into practice deftly and conscientiously.

Increasingly, publishers are encountering questions and demands from authors and academic editors regarding the inclusion and online posting of datasets and supplemental data. Major publishers are providing venues for the sharing of this information and taking into account that it too is being held up to scrutiny in a way that was not possible in the past.

There appeared to be agreement in many workgroups that the journal article and monograph are still the most important units of publishing—they have the most influence over promotion and tenure, the most impact on revenue for publishers, and are the most recognized forms of “true” scholarship for the majority of scholars. Nevertheless, there is also broad consensus that we need living, not static, documents; that a new generation of scholars is embracing digital humanities, computational sciences, and other new forms of scholarship; and that the focus on data and other research outputs is increasing. Most teams agreed that data, especially open data, is an increasing critical component not only of research but of dissemination, and that we need to improve ways of making data useful to others and preserving it for future use. Several workgroups expressed interest in raising the stature of and engagement with new scholarly artifacts (video, interactivity, social media, digital shorts, blog posts, metadata). Attendees were pleased to see fellow delegates share examples of that experimentation.

The Primacy of Promotion and Tenure

There was widespread agreement among many stakeholders represented at OSI that the promotion and tenure process is not evolving fast enough; in some respects promotion and tenure committees are often the most conservative stakeholders on many campuses, with built in incentives to keep things the way they always have been. While workgroups expressed a need to reform the promotion and tenure process, most P&T committees probably do not realize there is a problem that needs fixing.

Promotion and tenure was discussed at some point in most, if not all, workgroups. Notably, there was no team expressly designated to tackling the question of promotion and tenure (there were six groups, for example, with “open” expressly in the question or title) and there were no stakeholders specifically involved in promotion or tenure issues (although some delegates may have served at some point on P&T committees). Incentives for promotion and tenure most often do not align to open access goals, and the current reward system is resistant to change. There are misaligned incentives throughout the research publishing and distribution process, which several teams pointed out. Academic administrators were not well represented at these meetings (see discussion of stakeholders and missing voices, below).

Calls for “change” in the promotion and tenure process are legion. There are disciplinary differences as well as recognition that promotion and tenure processes and requirements differ greatly not only from one institution to another, but even among different schools and colleges at the same institution. There are widespread, and perhaps widening, differences between how digital humanities, digital publications, electronic only “shorts” and other products are treated or should be treated in promotion and tenure committees. Some delegates wondered what influence these conversations would have in the evolution of P&T practice. There is recognition that while promotion and tenure is a key component of the publishing ecosystem, there is perhaps little that publishers themselves can do to influence the process. In this sense, OSI could conceivably work with other stakeholders throughout the academic system to express perspectives and positions on this evolution.

It was suggested during our discussions that it might be productive to contact the organizations where Provosts and other academic officers discuss issues related to the faculty rewards system. Any movement to reform the P&T process in ways that would support Open initiatives would be likely to occur there. The Association of American Universities (AAU), Association of Public and Land-Grant Universities (APLU), and the recently formed Association of Chief Academic Officers (ACAO) are organizations where discussions about Promotion & Tenure are likely occurring. It could be productive for OSI to reach out to these organizations, find out whether these issues are currently on their agendas, and see if there are ways to work together to align the interests of OSI with any existing initiatives.

Stakeholders and Missing Voices

At-large delegates identified stakeholder groups whose voices were missing or underrepresented at the inaugural OSI meeting. Most of these stakeholder representatives were invited but did not attend, perhaps due to the time commitment involved or because the conference agenda did not align with their interests and roles. Missing voices included university research leadership (Provosts, deans, etc.). In many cases, these invited individuals ultimately decided to send librarians or people from the publishing arm of the organization, presumably because they believed these representatives to be more relevant to the discussion. The result is that the broader institutional perspective that these principals may have brought was lacking. A recommendation of the at-large workgroup is that OSI make a concerted effort to reach out to such constituencies in order to inform them of progress, rather than trying to bring them into the next meeting, considering that these leaders might not be able to provide the time commitment required by OSI participation.

A relative lack of geographic and demographic diversity was also noted. In future meetings, it would be desirable for OSI to integrate more perspectives from the Global South, Middle East, and East, as well as greater representation from traditionally underrepresented groups such as racial and ethnic minorities.

Some of the discussion focused on what would be good or useful for the researchers by those who are not necessarily researchers themselves (despite the fact that some delegates identified as representing two stakeholder groups, e.g., librarian/researcher, publisher/scholar). A much larger presence of scholars—researchers, investigators, and scientists—must be present for this discussion to have greater relevance. Again, however, the time commitment and perhaps a lack of strong interest in OA/publishing issues may continue to be a factor.

Another underrepresented faction at OSI included leading researchers in the science of science (scientific evaluation metrics). These researchers might have contributed to discussions on impact as well as the impact of academic culture on knowledge sharing.

More representation from non-governmental funding agencies and academic societies as membership organizations (not only as publishers) would also be helpful for future meetings.

Another issue that was generally not discussed (at least not mentioned in group presentations) was the rise of adjunct faculty, the (related) rise of scholars outside conventional academia, and the impact this may have on scholarly publishing. These faculty members and scholars may publish articles, monographs, and other research products because it’s good for their CV or personally rewarding, but they are not in the “publish or perish” system; these stakeholders may also assign fewer monographs in course syllabi. Thus, their presence certainly affects scholarly publishing but is frequently an overlooked “voice.”

The at-large workgroup also noted some challenges for OSI, both for this and future meetings related to balance of the stakeholder groups. Many perceived a heavy focus on STEM disciplines and relative lack of representation from the humanities and social sciences. Another challenge will be to reach common ground among diverse countries and disciplines.

While it will be a challenge to involve researchers, authors, academic administrators (e.g. provosts and deans) more directly, the lack of these voices may create misaligned goals, misunderstood future directions for OSI specifically, impracticable recommendations, and collaborative action that does not in fact consider diverse perspectives and spheres of influence within the scholarly publishing ecosystem.

The Influence of Impact

A theme running as an undercurrent in many workgroup discussions was a greater need to focus on assessment of the value of research and scholarship. For the “Impact Factors” workgroup, which some at-large members noted as one of the most collegial workgroups, this was the main theme of discussion, although they focused explicitly on the journal impact factor.

Notably, nearly all participants in the OSI2016 conference, and most stakeholders in the entire scholarly publishing ecosystem, have an interest and need to measure the impact of research and scholarship. For publishers, this involves both estimating in advance the possible impact in order to make publishing decisions and measuring its impact in order to communicate this to funders, shareholders, boards, authors, and others. For academic libraries, this involves communicating impact to faculty and administrators as well as providing faculty, students, and others with tools to measure or evaluate impact and value on their own. Funders are interested in the value of their investment for myriad reasons. Researchers and scholars are vested with communicating the impact of their research to promotion and tenure committees, funders, and other stakeholders, while considering the impact of the research of other scholars in their field. Administrators of universities and research institutes need to evaluate the impact of the research of individual scholars, research teams and departments, and assess the impact of the institution as a whole, often as a component of strategic plans, annual reports, or other communications. But precisely how to measure the value and impact of research and its dissemination remains challenging, as it is widely recognized that current means of measurement (both traditional and alternative) are imperfect, subject to manipulation, and often imprecise.

While many if not most OSI2016 delegates assume a priori that open access has a positive effect on the impact of research, this is an area that needs more evaluation.

Much work must occur, and undoubtedly will occur, to improve methodologies, tools, and rubrics for evaluating and communicating the value and impact of research. What will doubtlessly remain a challenge will be measuring the true impact of research on policy, society, and quality of life.

We can extend this need to measure the impact of OSI itself.

Moving Forward: OSI2017 and Beyond

As at-large delegates, we noted that enthusiasm for this initiative was universal among the workgroup participants. People came committed to the initiative’s goals, they participated, and they worked! (They seemed to have fun too.)

As discussed above, some significant stakeholder representatives (including provosts, deans, senior researchers, and authors) did not attend or were underrepresented, perhaps because of the time commitment. Researchers themselves might not have appreciated the importance of taking a seat at this table. For many, the subject matter is daunting and would have taken valuable time away from their research projects and academic commitments at a particularly busy time of the academic year. Some invited delegates may ultimately not have attended due to the voluminous and sometimes contentious pre-conference listserv discussions. These issues need to be resolved going forward (as of this writing, the voluminous listserv discussions have not abated—there must be a way to be involved in the thrust of conference and not involved in every aspect or permutation of such discussions).

The need for collective action is recognized, as is the need to tell a coherent story. Crowd share or other collective action tools should be explored to drive this discussion forward and further. OSI both recognizes and contributes to the need for better ways of communication and idea-sharing between different sectors of the scholarly publishing ecosystem, of ways to tell or share the story of open access collectively to specific audiences in academia and beyond.

Most delegates and teams came with the assumption that OA is the best goal for future. Some discussed this but perhaps didn’t question the assumption enough, but open is more complicated (as evidenced by the number of workgroups with “Open” in the title). Several delegates mentioned a need for more research on the impact of OA (crucially, does OA contribute directly to better research or better outcomes?).

At-large delegates discussed the possible focus of the next OSI meeting. Questions include whether every topic requires another study before the next OSI meeting (many workgroups argued for the need for further research) and whether new standards need to be developed (ideas included a “basket of impact metrics” and an “openness score”).

Many delegates expressed a desire to move the discussion forward and accomplish something concrete (but what that is, however, remains to be clearly defined). Some delegates expressed confusion about the ultimate goal of this meeting. How can future OSI meetings and their delegates deliver tangible products or initiatives? How can we measure progress toward goals? What do we hope to achieve over the span of the planned 10 years, how do we get there, and what would the results look like?

At the final presentations on Friday morning, many of the workgroups attempted to offer concrete proposals for work that can be undertaken over the next year. How people will be organized to follow up on those recommendations remains unclear. An organizing structure for OSI will need to be defined if more is to be accomplished in the next 12 months than planning for another conference.

OSI2016 At-Large Workgroup

Ali Andalibi, Associate Dean of Research, College of Science, George Mason University
Todd Carpenter, Executive Director, National Information Standards Organization (NISO)
Sindy Escobar-Alvarez, Senior Program Officer, Doris Duke Charitable Foundation Medical Research Program
Steve Fiore, President, Interdisciplinary Network for Group Research (INGRoup)
William Gunn, Director of Scholarly Communications, Elsevier
Nancy Gwinn, Director, Smithsonian Libraries
Norbert Lossau, Vice President, University of Göttingen
Judy Luther, President, Informed Strategies
Dave McColgin, User Experience Director, Artefact
Meredith Morovati, Executive Director, Dryad
Scott Plutchak, Director of Digital Data Curation Strategies, University of Alabama at Birmingham
Jessica Sebeok, Associate Vice President for Policy, Association of American Universities (AAU)
Concetta Seminara, Editorial Director, US Social Science & Humanities Journals Program, Routledge/Taylor & Francis
Susan Skomal, President/CEO, BioOne
Mark Ware, Director, Mark Ware Consulting
John Warren, Head, Mason Publishing Group, George Mason University Press, George Mason University
Mary Woolley, President, Research!America
Grace Xiao, Co-Founder and President, Kynplex
John Zenelis, Dean of Libraries and University Librarian, George Mason University