Helping science succeed
Helping science succeed

Report from the At-large Workgroup

Download

Abstract / OSI2016 Workgroup Question

The At-Large workgroup was the largest and most diverse in terms of stakeholder representa­tion. At-large delegates observed workgroup conversations during the meeting and contributed to these conversations while letting the workgroup teams answer their questions. At-large delegates met during the conference and convened online both before and after the conference in order to develop some high level takeaways based on what they observed. While workgroups focused on answering a particular big question confronting scholarly publish­ing, at-large delegates had a wide-angle lens on the evolution of these questions and proceedings.

Introduction

Members of the At-Large workgroup had the distinct pleasure of attending and ob­serving a variety of other workgroup sections. Some at-large delegates partici­pated during the entire Open Scholarship Initiative (OSI) 2016 meeting while others attended only a few sessions. Neverthe­less, this allowed us to gain a global picture of the process.

Collectively we noted some themes and observations during OSI2016. The observa­tions we present in this paper reflect a variety of viewpoints—including our individual observations of group pro­cess and themes, comments made to at-large delegates during sessions and breaks, and our reflections after the fact.

Observations on the Format and Process of OSI2016

The Open Scholarship Initiative set as its goal the creation of a framework for discus­sion and collaborative action between stakeholders from diverse perspec­tives and spheres in the scholarly publishing ecosystem. Workgroups met several times over the course of two days for at least eight hours of face-to-face conversation. Each workgroup was tasked with creating a preliminary presentation (during which feedback from other dele­gates was offered) and a final presentation; teams were also tasked with the creation of a final paper to be com­pleted within two weeks of the close of the conference.

Facilitation training was offered but in general facilitators were not determined in advance; instead, the process of selecting facilitators or speakers unfolded organ­ically and differed from group to group. The facilitator training provided was help­ful even if all workgroups may not have followed these best practices. Even so, spokespersons, collaboration, and consen­sus emerged throughout each workgroup discussion.

Ground rules were limited; the most im­portant rule, according to the organizing committee, was to be “open and respect­ful.” Other suggestions, albeit not enforced, included the establishment of workgroup-specific ground rules; limiting side conversations and interruptions; and refraining from tweets or other postings about group process and attributions of statements from individual workgroup members.

Formation of workgroups around specific themes was helpful in bringing together individuals with similar interests, if not necessarily similar viewpoints. The pro­cess evolved and played out distinctly in different groups. Many teams referred to the guidelines for facilitators, which prompted activities and proposed out­comes for each workgroup session. Some delegates nevertheless felt that more struc­ture would have been helpful to direct the conversation and enhance productivity; it is unclear whether these delegates had taken or had referred to the facilitation training. The topics ranged in focus and scope, some being more practical and answerable than others. In some groups, opposing philosophical positions—for example on moral issues surrounding open access (OA)—provoked lively de­bates that generally resulted in reaching some common ground or at least respect for others’ viewpoints. To some delegates, however, this appeared to be less an oppor­tunity to discover common ground or achieve consensus than setting up bar­riers to further discussion. For example, one at-large delegate noted discord among members of a workgroup regarding how OA is mandated in various countries and geographic regions or across diverse aca­demic disciplines; perspectives of OA among commercial publishers, university presses, libraries, and institutional and government repositories; and commercial versus humanitarian goals. This at-large delegate made a comment during the team’s discussion about these differences and was met with negativity from a mem­ber of the workgroup. Nonetheless, most at-large delegates were welcomed as observ­ers in the groups they visited, and were often able to provide perspectives that were useful to group work, by offer­ing comments such as “have you considered this (x) aspect?”

It may have been helpful for members of each workgroup to prepare a literature review relevant to each conversation prior to the meeting. Too little reference was made in some workgroup discussions and presentations to relevant existing work, projects, evidence, or data. Five tutorials were offered to delegates prior to the con­ference to provide a common background on the issues to be discussed. Some dele­gates expressed the reaction that there was an overabundance of documentation, very little of which was possible to ingest for those with day-jobs. This was particularly true of email discussions, which clearly overwhelmed many delegates. Common expressions were of the nature of “I just tuned everything out because it was too much.” Laying some groundwork ahead of the meeting, including consideration of existing initiatives and results would help ensure that the teams built on previous work by others. More broadly, OSI should consider how it is positioned and differen­tiated in the landscape of existing initiatives at national and international levels, and should highlight the unique position of OSI.

At-large members also observed that in many cases, workgroups may have “ta­bled” topics that important, because the issues were too “big” and therefore not able to be thoughtfully addressed in the forum. Workgroup topics frequently over­lapped, but a particular workgroup was unsure whether they should address “x” topic because “x” was probably being covered by another team. It was re­marked, for instance, that open data is different than open access; this important element was perhaps not thought through as much as it could have been.

At-large delegates noted that providing feedback on individual workgroups was somewhat thwarted by the individual groups’ formats. Most at-large members visited as many workgroups as possible, thereby only visiting each one for a few hours. This provided a snapshot of topics and process, but did not elicit a meaning­ful understanding of how any one particular group performed. In addition, the workgroups themselves ranged in topic focus from very general to much more specific, so evaluation was not overly supported. Workgroup discussions and presentations ranged from “big pic­ture” to practical, and from pragmatic to aspirational goals. The delegate-at-large role could have been more specifically formulated. Lack of focus can unintention­ally present issues when synthesizing and analyzing experience. This may have yielded discussions that echoed outcomes from other conferences, workshops, or meetings, without resulting in any new findings or recommendations.

At-large delegates noted a considerable, if not surprising, overlap in themes. Some workgroups and individual delegates re­marked that a greater awareness of the progress and thrust of other workgroups would have been helpful; some indicated that it seemed as if teams were operating in isolation and delving into ideas and questions already being considered by other workgroups. The preliminary presen­tations addressed this need but may have come too late for some; it’s difficult to consider other ways this could have been addressed, however, under the time constraints of this conference. In hind­sight, summarizing the groups’ work through presentations on the second day of the meeting proved sufficient, thus final presentations on the third day were perhaps not as fruitful; some delegates expressed the opinion that the Friday morning session was overkill. This final session time, when less energy was avail­able and some delegates had started to leave, could have been more productively used as an open forum and extended discus­sion on moving forward.

Changing Scholarly Outputs and Changing Needs

Among the most common themes of the conference, discussed throughout most of the workgroup sessions that at-large dele­gates observed, were the changing nature of scholarly “publishing” products and how these in turn both reflect and affect the changing needs of the academy. Chang­ing scholarly outputs include products of the digital humanities and other digital publication; the emergence of non-linear narrative formats and new forms of storytelling; data publication and visualization; and other research outputs. Comments and discussions centered on changes to the peer review process; the need for change in the promotion and tenure process at most universities; a need for more descriptive and inclusive credits beyond authors and acknowledgements; how this affects decisions on how to cu­rate, cite, preserve, and normalize data; and how the impact of these varying forms of scholarship is measured.

In many workgroups, these issues of “non­traditional” scholarship, digital publications, and digital humanities prod­ucts were considered significant but somewhat off the main thrust of the workgroup’s agenda, and thus, con­strained by time and the need to create a cohesive presentation in two days, these issues were often placed in the “parking lot.” In other groups, such as “What is Publishing (1)” and “Access and Preser­vation,” these issues influenced the thrust of the presentation even if, ultimately and not surprisingly, “answers” to these issues were not provided.

From the perspective of scholarly pub­lishers, librarians, and other stakeholders represented at OSI, there is an eagerness to bring about an evolution of peer review as it develops in the 21st century. Ques­tions surrounding peer review emerged at some point in the process during most team discussions, not only in the “Peer Review” workgroup. Questions raised frequently included whether peer reviews should be open, whether peer reviewer names should be made available, the role of post-publication peer review, peer re­view of data, and so on. There is agreement that the perceived and possibly ageist, sexist, and racist biases that may have pervaded the peer review process for hundreds of years are finally being ad­dressed. Publishers need to be prepared to accept new methods and assist in putting them into practice deftly and conscien­tiously.

Increasingly, publishers are encountering questions and demands from authors and academic editors regarding the inclusion and online posting of datasets and supple­mental data. Major publishers are providing venues for the sharing of this information and taking into account that it too is being held up to scrutiny in a way that was not possible in the past.

There appeared to be agreement in many workgroups that the journal article and monograph are still the most important units of publishing—they have the most influence over promotion and tenure, the most impact on revenue for publishers, and are the most recognized forms of “true” scholarship for the majority of scholars. Nevertheless, there is also broad consensus that we need living, not static, documents; that a new generation of schol­ars is embracing digital humanities, computational sciences, and other new forms of scholarship; and that the focus on data and other research outputs is in­creasing. Most teams agreed that data, especially open data, is an increasing criti­cal component not only of research but of dissemination, and that we need to im­prove ways of making data useful to others and preserving it for future use. Several workgroups expressed interest in raising the stature of and engagement with new scholarly artifacts (video, interactivity, social media, digital shorts, blog posts, metadata). Attendees were pleased to see fellow delegates share examples of that experimentation.

The Primacy of Promotion and Tenure

There was widespread agreement among many stakeholders represented at OSI that the promotion and tenure process is not evolving fast enough; in some respects promotion and tenure committees are often the most conservative stakeholders on many campuses, with built in incen­tives to keep things the way they always have been. While workgroups expressed a need to reform the promotion and tenure process, most P&T committees probably do not realize there is a problem that needs fixing.

Promotion and tenure was discussed at some point in most, if not all, workgroups. Notably, there was no team expressly designated to tackling the ques­tion of promotion and tenure (there were six groups, for example, with “open” ex­pressly in the question or title) and there were no stakeholders specifically involved in promotion or tenure issues (although some delegates may have served at some point on P&T committees). Incentives for promotion and tenure most often do not align to open access goals, and the current reward system is resistant to change. There are misaligned incentives through­out the research publishing and distribution process, which several teams pointed out. Academic administrators were not well represented at these meet­ings (see discussion of stakeholders and missing voices, below).

Calls for “change” in the promotion and tenure process are legion. There are discipli­nary differences as well as recognition that promotion and tenure processes and requirements differ greatly not only from one institution to another, but even among different schools and colleges at the same institution. There are widespread, and perhaps widening, differ­ences between how digital humanities, digital publications, electronic only “shorts” and other products are treated or should be treated in promotion and tenure committees. Some delegates wondered what influence these conversations would have in the evolution of P&T practice. There is recognition that while promotion and tenure is a key component of the pub­lishing ecosystem, there is perhaps little that publishers themselves can do to influ­ence the process. In this sense, OSI could conceivably work with other stakeholders throughout the academic system to ex­press perspectives and positions on this evolution.

It was suggested during our discussions that it might be productive to contact the organizations where Provosts and other academic officers discuss issues related to the faculty rewards system. Any move­ment to reform the P&T process in ways that would support Open initiatives would be likely to occur there. The Association of American Universities (AAU), Associa­tion of Public and Land-Grant Universities (APLU), and the recently formed Association of Chief Academic Officers (ACAO) are organizations where discussions about Promotion & Tenure are likely occurring. It could be productive for OSI to reach out to these organiza­tions, find out whether these issues are currently on their agendas, and see if there are ways to work together to align the interests of OSI with any existing initia­tives.

Stakeholders and Missing Voices

At-large delegates identified stakeholder groups whose voices were missing or un­derrepresented at the inaugural OSI meeting. Most of these stakeholder repre­sentatives were invited but did not attend, perhaps due to the time commitment in­volved or because the conference agenda did not align with their interests and roles. Missing voices included university re­search leadership (Provosts, deans, etc.). In many cases, these invited individuals ultimately decided to send librarians or people from the publishing arm of the organization, presumably because they believed these representatives to be more relevant to the discussion. The result is that the broader institutional perspective that these principals may have brought was lacking. A recommendation of the at-large workgroup is that OSI make a con­certed effort to reach out to such constituencies in order to inform them of progress, rather than trying to bring them into the next meeting, considering that these leaders might not be able to provide the time commitment required by OSI participation.

A relative lack of geographic and demo­graphic diversity was also noted. In future meetings, it would be desirable for OSI to integrate more perspectives from the Global South, Middle East, and East, as well as greater representation from tradi­tionally underrepresented groups such as racial and ethnic minorities.

Some of the discussion focused on what would be good or useful for the research­ers by those who are not necessarily researchers themselves (despite the fact that some delegates identified as represent­ing two stakeholder groups, e.g., librarian/researcher, publisher/scholar). A much larger presence of scholars—research­ers, investigators, and scientists—must be present for this discussion to have greater relevance. Again, however, the time commitment and perhaps a lack of strong interest in OA/publishing issues may continue to be a factor.

Another underrepresented faction at OSI included leading researchers in the science of science (scientific evaluation metrics). These researchers might have contributed to discussions on impact as well as the impact of academic culture on knowledge sharing.

More representation from non-govern­mental funding agencies and academic societies as membership organizations (not only as publishers) would also be helpful for future meetings.

Another issue that was generally not dis­cussed (at least not mentioned in group presentations) was the rise of adjunct fac­ulty, the (related) rise of scholars outside conventional academia, and the impact this may have on scholarly publishing. These faculty members and scholars may publish articles, monographs, and other research products because it’s good for their CV or personally rewarding, but they are not in the “publish or perish” system; these stakeholders may also assign fewer monographs in course syllabi. Thus, their presence certainly affects scholarly publish­ing but is frequently an overlooked “voice.”

The at-large workgroup also noted some challenges for OSI, both for this and fu­ture meetings related to balance of the stakeholder groups. Many perceived a heavy focus on STEM disciplines and relative lack of representation from the humanities and social sciences. Another challenge will be to reach common ground among diverse countries and disci­plines.

While it will be a challenge to involve re­searchers, authors, academic administrators (e.g. provosts and deans) more directly, the lack of these voices may create misaligned goals, misunderstood future directions for OSI specifically, im­practicable recommendations, and collaborative action that does not in fact consider diverse perspectives and spheres of influence within the scholarly publish­ing ecosystem.

The Influence of Impact

A theme running as an undercurrent in many workgroup discussions was a greater need to focus on assessment of the value of research and scholarship. For the “Im­pact Factors” workgroup, which some at-large members noted as one of the most collegial workgroups, this was the main theme of discussion, although they fo­cused explicitly on the journal impact factor.

Notably, nearly all participants in the OSI2016 conference, and most stakehold­ers in the entire scholarly publishing ecosystem, have an interest and need to measure the impact of research and scholar­ship. For publishers, this involves both estimating in advance the possible impact in order to make publishing deci­sions and measuring its impact in order to communicate this to funders, sharehold­ers, boards, authors, and others. For academic libraries, this involves communi­cating impact to faculty and administrators as well as providing faculty, students, and others with tools to measure or evaluate impact and value on their own. Funders are interested in the value of their investment for myriad reasons. Research­ers and scholars are vested with communicating the impact of their re­search to promotion and tenure committees, funders, and other stakehold­ers, while considering the impact of the research of other scholars in their field. Administrators of universities and re­search institutes need to evaluate the impact of the research of individual schol­ars, research teams and departments, and assess the impact of the institution as a whole, often as a component of strategic plans, annual reports, or other communica­tions. But precisely how to measure the value and impact of research and its dissemination remains challenging, as it is widely recognized that current means of measurement (both traditional and alternative) are imperfect, subject to manipulation, and often imprecise.

While many if not most OSI2016 dele­gates assume a priori that open access has a positive effect on the impact of research, this is an area that needs more evaluation.

Much work must occur, and undoubtedly will occur, to improve methodologies, tools, and rubrics for evaluating and com­municating the value and impact of research. What will doubtlessly remain a challenge will be measuring the true im­pact of research on policy, society, and quality of life.

We can extend this need to measure the impact of OSI itself.

Moving Forward: OSI2017 and Beyond

As at-large delegates, we noted that enthusi­asm for this initiative was universal among the workgroup participants. People came committed to the initiative’s goals, they participated, and they worked! (They seemed to have fun too.)

As discussed above, some significant stake­holder representatives (including provosts, deans, senior researchers, and authors) did not attend or were underrepre­sented, perhaps because of the time commitment. Researchers them­selves might not have appreciated the importance of taking a seat at this table. For many, the subject matter is daunting and would have taken valuable time away from their research projects and academic commitments at a particularly busy time of the academic year. Some invited dele­gates may ultimately not have attended due to the voluminous and sometimes contentious pre-conference listserv discus­sions. These issues need to be resolved going forward (as of this writing, the voluminous listserv discussions have not abated—there must be a way to be involved in the thrust of conference and not involved in every aspect or permuta­tion of such discussions).

The need for collective action is recog­nized, as is the need to tell a coherent story. Crowd share or other collective action tools should be explored to drive this discussion forward and further. OSI both recognizes and contributes to the need for better ways of communication and idea-sharing between different sectors of the scholarly publishing ecosystem, of ways to tell or share the story of open access collectively to specific audiences in academia and beyond.

Most delegates and teams came with the assumption that OA is the best goal for future. Some discussed this but perhaps didn’t question the assumption enough, but open is more complicated (as evi­denced by the number of workgroups with “Open” in the title). Several delegates mentioned a need for more research on the impact of OA (crucially, does OA contribute directly to better research or better outcomes?).

At-large delegates discussed the possible focus of the next OSI meeting. Questions include whether every topic requires an­other study before the next OSI meeting (many workgroups argued for the need for further research) and whether new standards need to be developed (ideas included a “basket of impact metrics” and an “openness score”).

Many delegates expressed a desire to move the discussion forward and accom­plish something concrete (but what that is, however, remains to be clearly defined). Some delegates expressed confusion about the ultimate goal of this meeting. How can future OSI meetings and their delegates deliver tangible products or initia­tives? How can we measure progress toward goals? What do we hope to achieve over the span of the planned 10 years, how do we get there, and what would the results look like?

At the final presentations on Friday morn­ing, many of the workgroups attempted to offer concrete proposals for work that can be undertaken over the next year. How people will be organized to follow up on those recommendations remains unclear. An organizing structure for OSI will need to be defined if more is to be accom­plished in the next 12 months than planning for another conference.

OSI2016 At-Large Workgroup

  • Ali Andalibi, Associate Dean of Research, College of Science, George Mason University
  • Todd Carpenter, Executive Director, National Information Standards Organization (NISO)
  • Sindy Escobar-Alvarez, Senior Program Officer, Doris Duke Charitable Foundation Medical Research Program
  • Steve Fiore, President, Interdisciplinary Network for Group Research (INGRoup)
  • William Gunn, Director of Scholarly Communications, Elsevier
  • Nancy Gwinn, Director, Smithsonian Libraries
  • Norbert Lossau, Vice President, University of Göttingen
  • Judy Luther, President, Informed Strategies
  • Dave McColgin, User Experience Director, Artefact
  • Meredith Morovati, Executive Director, Dryad
  • Scott Plutchak, Director of Digital Data Curation Strategies, University of Alabama at Birmingham
  • Jessica Sebeok, Associate Vice President for Policy, Association of American Universities (AAU)
  • Concetta Seminara, Editorial Director, US Social Science & Humanities Journals Program, Routledge/Taylor & Francis
  • Susan Skomal, President/CEO, BioOne
  • Mark Ware, Director, Mark Ware Consulting
  • John Warren, Head, Mason Publishing Group, George Mason University Press, George Mason University
  • Mary Woolley, President, Research!America
  • Grace Xiao, Co-Founder and President, Kynplex
  • John Zenelis, Dean of Libraries and University Librarian, George Mason University