The Journal of Digital Humanities: Post-Publication Review or the Worst of Peer Review?

peerreview

**Updated September 16: I am grateful to JDH for publishing these posts along with various responses. Soon after my posts were published, JDH approached Roopika and I to ask if we would approve publishing all the correspondence, with identifying information redacted. We asked our contributors for their opinions, and as a group came to the decision that it would fracture our field further to make all the correspondence public.

We offered to explore the option of co-publishing a post on best practices and lessons learned with experimental forms of publication with JDH, but the option was declined. However, we remain hopeful that this option to work with JDH to work on a co-authored piece may later become available. Thank you all for weighing in. 

 

For those just jumping into the discussion, I’ve published a follow-up blog post here, “Crowdsourcing Best Practices for Experimental Practices: Transparency.” I hope you will join in and offer suggestions. 

The problems of traditional peer review are well known. Peer review is not transparent; it takes too long; the true blindness of peer review is questionable, especially in small fields; its gatekeeping function encourages the conservatism of scholarship. To address these concerns, the Journal of Digital Humanities published its first issue in Winter 2011 as an effort to work through some of these problems. JDH champions a “post-publication” model of review where its editors collate some of the best existing DH work and publish it in journal form. How well does this publication model work in practice?

As my collaborator Roopika Risam and I discovered, the JDH model is far from transparent and, if anything, can intensify some of the problems of traditional peer review by inflating expectations around transparency. While working with JDH to publish a special section on the postcolonial digital humanities (#dhpoco), we learned that:

  1. The journal promised us one process for working with them, and once we had submitted everything, presented a completely new one; and that

  2. This sudden about-turn suggests that the journal is not fulfilling its mission to offer an improvement over traditional peer review.

Background

Roopika Risam and I approached JDH to publish a special section on the postcolonial digital humanities (#dhpoco) in July. They appeared enthusiastic, asking us to aim for a very ambitious four-week deadline.  It was agreed that we had four weeks to solicit revised versions of already published material to be submitted to JDH in four weeks for line editing and minor proofreading. Three weeks after submission the special section was to be published. (See the images at the end of the post for the agreed-upon process.) The timeline was short, but we pushed our authors to get the work done in time, and they did.

Five days after submitting our special section, Roopika and I were surprised to receive an email saying that the journal suddenly wanted us to submit our special section to a blind external review. On the one hand, we would have been glad for review, knowing that it would only make our work better. But upon further discussion, we started thinking about the following:

  1. Why was this request not made up front? I have had multiple dealings with special sections/issues of peer reviewed journals, and if something is marked as such, it does not go through external peer review unless this is requested at the outset.

  2. Since JDH emphasizes a “post-publication” model, and our essays had already been available in blog posts and forum responses, didn’t our editorial process serve as the second round of collation and post-publication–as the journal claims is its experimental role? From speaking with other special section editors, none had undergone a similar process. One person even related that she received an email from JDH asking to publish her already-published blog post in a week, and if she would make revisions that would be great, if not, they would publish it anyway.

I raise these claims to highlight JDH’s apparent lack of transparency on the following issues:

  • We appeared to be the only special section ever singled out for external peer review

  • JDH promised us one process, but then insisted upon another one

  • All of our essays had been published on the Internet through blog posts, the #dhpoco website and the summer school. This meant that any external review could hardly be blind. Why did JDH think that this would be blind, given that names and previous forms of our essays were already freely available online?

  • If blind peer review was not possible, why did JDH want to appoint a blind reviewer? Since it is an experimental journal, we would have been amenable to an open peer review.

Is JDH’s Post-Publication Model an Improvement Upon The Current Model for Peer Review?

This leads me to the following question: Is JDH’s post-publication model an improvement over the existing peer review model?

  • Our experience with JDH has been anything but transparent. How, then, does this journal further claims about making the process of peer review more transparent?

  • JDH seems to publish work by the same people, multiple times–and from what I have heard anecdotally, they have been asked for minimal revision, and for no external review. Do our experiences indicate a–hopefully unconscious–bias within the journal?

  • If this bias exists, how is JDH’s model an improvement over traditional peer review? What is the process that determines what posts needs to go through external review, and what doesn’t?

  • Has JDH put any measures in place to ensure that its policies are transparent, fair and unbiased?

These types of problems are not exclusive to JDH. They concern any publication offering an alternative model to peer review. I believe that we do need to fix the existing issues within the system of peer review. But if what we experienced is an alternative model, it does not offer an improvement over the original system. This is even more disturbing because by being a pioneer in alternative forms of peer review, JDH could serve as a role model for other outlets wanting to experiment.

These new experimental journals will need to find a way to address these problems. By suddenly changing their process, JDH wasted a lot of our time and the time of our contributors, who had to rush to put the pieces together to meet their deadline. If we had known about the external review to begin with, we would have not had to rush our writers. Their sudden reversal broke our trust in their process, a necessary quality to have for any new form of peer review to work.

I hope that I am mistaken about JDH. Again, I am sure that our essays–as would every piece of work that has already been published–would benefit from review. What I am not sure about, however, is why we were singled out for special treatment, and why the journal had not been transparent with us from the beginning. This makes me worry about the new forms of review that the journal claims to champion and about the unseemly ways it can be put to use. However, we do believe that the review process is a useful one. We are taking our special section to another venue where it will be peer reviewed.

Going forward, I hope that JDH will consider making its measures for fairness and transparency public, and to implement some checks and balances to provide some accountability for these measures. We need to do better.

 

Have you ever worked with JDH? We’re collecting data about different experiences on working with the journal. Please fill out the form and send it to as many people as you know.

** I thank Roopika Risam, Natalia Cecire, Alex Gil, Jesse Stommel and Fiona Barnett for their comments on my previous draft.

Responses to this Post

Previous Correspondence With JDH

Top image credit

32 Responses to “The Journal of Digital Humanities: Post-Publication Review or the Worst of Peer Review?”

  1. James O'Sullivan August 29, 2013 at 10:05 am #

    Interesting article. With the following point, I think you raise a concern that I have also tried to raise on occasion:

    “JDH seems to publish work by the same people, multiple times–and from what I have heard anecdotally, they have been asked for minimal revision, and for no external review. Do our experiences indicate a–hopefully unconscious–bias within the journal?”

    I have found this to be the case at particular conferences, where, in the absence of an appropriately transparent and blind review process, scholars from particular communities are being given precedence over those coming from “the outside”. I recently attended a conference where (as far as I could tell) 90% of postgrads from a particular cohort had papers accepted, and a senior scholar from the same community was named on no less than seven papers.

    Digital humanities, as an emerging field, needs to avoid pockets forming within which groups of scholars, who are undoubtedly pioneers and highly respected, dominate proceedings. When a peer review process is appropriately transparent and blind then this is fine – but when it isn’t, it raises serious concerns over the validity of scholarship, many of which are most likely unwarranted, but nonetheless a natural reaction. You don’t think, that’s there because of merit; you think, it’s there because of who wrote it.

  2. Jesse Stommel (@Jessifer) August 29, 2013 at 10:52 am #

    Adeline, what I find most valuable is how open and transparent you have been in publishing this piece: inviting peers to review several drafts openly via Google Docs and Facebook, offering JDH an opportunity to respond before publication, avoiding premature conclusions to encourage conversation post-publication, etc. It’s an important discussion, not just about JDH, but about academic publishing and review more broadly.

  3. Scott Weingart August 29, 2013 at 11:46 am #

    As a guest editor of a previous issue of the JDH, I’d like to add my perspective to the mix. We encountered a similar – but not exactly the same – environment as Adeline and Roopika; the editors seemed to be making things up as they went along, and we didn’t really have a clear list of policies from the beginning. For our issue, we considered this a strength; it allowed us to work with the editors and authors to make a format that was well-suited for our topic. I’d considered this approach impressive in its flexibility, and was thankful for it.

    After reading Adeline’s post, I can see that she did not have the same experience, and in retrospect my guess is that if it hadn’t been her and Roopika on #dhpoco, it would have been some other group to have encountered these problems somewhere down the line. The same flexibility that Elijah and I enjoyed can swing in the opposite direction if there is poor communication about expectations along with tight deadlines, as in this case there were. I think the guest editors did the right thing in discussing this publicly, and hopefully the JDH group specifically and the DH community in general will learn from it and evolve from here on out.

    This is the danger of the pendulum swinging to a little bit *too* flexible. Perhaps the journal can now institute a policy where the editors, guest editors, and perhaps a rotating panel from the community can have a brief period to set and agree on the editorial policy for the forthcoming issue, thus allowing JDH to continue to be flexible and evolving, but making sure serious issues like these don’t crop up again.

    • Natalia August 29, 2013 at 12:04 pm #

      Scott, I think you may be right that this “flexible” (haphazard?) process was a problem waiting to happen. The question is, why did it happen to these particular writers? Was it just chance?

      You can sit perfectly still in a train and still be moving very fast. There doesn’t have to be a conscious intention to exclude for defaults to be powerful, and the flexibility that was so useful for you and Elijah may be the reverse for others, not just randomly, but in systematic ways. Not to put too fine a point on it: ad-hoc, “flexible” procedures often benefit people whom the system already endorses.

      As you say, measures need to be put in place to prevent problems like this.

    • James O'Sullivan August 29, 2013 at 12:14 pm #

      Scott, you raise some very interesting points here, and for the most part I’d wholeheartedly agree – the scope for evolution and flexibility is there as a product of the JDH model, and this is positive.

      However, I also have to agree with Natalia, when she suggests that this approach can lead to benefiting “people whom the system already endorses”.

      I don’t think that the problem is whether or not this has actually been the case – most of the articles that I’ve encountered in JDH (to which I’ve never submitted) were of a high standard, but the issue is one of credibility. If there is any doubt whatsoever within a selection and peer-review process, there will always be doubts, particularly in such a subjective field. It’s not scholarly for names to go before content, BUT, nor is it fair for respected scholars to have their work viewed with suspicion, when it’s most likely quality, rather than favouritism, that has secured its publication. But as long as there is doubt, this will be an unfortunate consequence.

      Essentially, it’s a question of removing any such doubt.

    • Scott Weingart August 30, 2013 at 11:05 am #

      That’s fair points, both, and put much better than I ever could. Still, it feels like the discussion surrounding this issue has focused on what went wrong, rather than how to fix it, which ultimately is where we need to be.

  4. Siobhan Senier August 29, 2013 at 1:43 pm #

    This is so unfortunate. A flexible and contingent editorial policy is, yes, probably ill-advised; when the bar is suddenly raised for two guest editors who just “happen to be” junior women of color, you have a pretty damning embarrassment on your hands. It doesn’t matter whether this is intentional or not (and to be honest, it’s kind of boring to have to point that out. That’s how racism and similar forms of exclusion work: systemically. We’re supposed to be thoughtful enough at least to TRY guard against it, through constant self-questioning and conferring with others.)

    Your post made me look more closely at JDH’s editorial staff, and I find myself wondering how many other journals—new or old, experimental or otherwise—are being run solely by assistant professors with relatively short publishing experience. I hope it’s clear that I am not slighting junior scholars (whose expertise—like yours and Roopika’s—is essential, especially in emerging fields), just noting what looks like a bit of an anomaly. If JDH does have a larger advisory board, perhaps one that includes more senior and more-published scholars, those names do not seem to be on their website. If, however, it is placing the entire burden of editing on two junior scholars, who are undoubtedly overworked and overstressed, and adding editorial “flexibility” into the mix, that seems like a definite problem with resource allocation—unfair, really.
    To everyone involved.

  5. mnsmith August 29, 2013 at 2:26 pm #

    What I find most disturbing here is that the “rules of the game” seem to have changed while the game was being played; that JDH appears to be changing its story; that “blind” review is being proposed for a journal touting post-publication review; that the “blind” review is one-sided. There is no way a critical reviewer would not know who wrote the essays, so the review would not be blind from the reviewer’s standpoint. What then, should JDH do? In my opinion, the critical reviews should not be anonymous just as the authors are not anonymous. In the world of Emily Dickinson, I have started signing ALL of my book and article reviews, and in the DH and other worlds where I review, I always say that an editor or conference director can let the authors know who reviewed their piece. If I’m going to pass judgment that may have direct impact on someone’s career, why should it be otherwise?

    By the way, I started doing this because another Dickinson critic started telling everyone that I was “out to get her” and started saying I was the reviewer for every book proposal, book, article, and conference proposal she submitted that got rejected. The only time I have reviewed one of her pieces was a book and I recommended “revise and resubmit.” But the wild accusations made me angry, and so I went to being an open reviewer. I also have done this in renewal of contract, tenure, and promotion decisions–if I vote against someone or abstain (which I never do, by the way), I tell the person in question. This caused some short-term discomfort, but I’ve found that candidates and authors appreciate my being forthcoming and I’ve also found that it’s led to very important and productive conversations. Without being so open, there’s always ridiculous speculation–which three do you think voted against me? don’t you think that he voted against her? is she trying to sabotage him? and so forth and so on. Anonymity has made us weaker, and confidentiality in our fields is nigh unto laughable. It’s rarely needed, though it’s invoked all the time when people really mean secrecy.

    So has JDH said why they wouldn’t be willing to let you have a REAL open critical review?

    Thanks for your bravery in making this public. Usually such things just go on because folks are afraid of retribution. You’re helping create a world in which we can have real exchanges of disagreement and where real consensus might be forged (rather than coerced consensus based on secrecy). Let’s hope JDH takes up your implicit offer of open critical review and doesn’t just continue in a defensive mode (I find that disturbing too).

    Thank you for your candor and most of all for your first-rate work!
    –Martha Nell Smith

  6. Timothy Burke August 29, 2013 at 3:28 pm #

    So on one hand I would hate it if the outcomes of expressing these kinds of concerns were more rigid formalizations (especially if the formalizations weren’t themselves defended or described with some degree of transparency). I like DH environments where there is some willingness to experiment with form, process and norms and where a certain of communicative informality is encouraged. But the point it seems to me is that this spirit only works if there is a relatively ‘flat’ collaborative environment and if everyone puts their cards on the table at all times, and if there is an active embrace of the autonomy of collaborators to try what they want to try.

    Meaning, if you hand someone the keys to a shared resource or platform that you are the steward for and say, “Do it your way”, you stick to that even if what it produces isn’t your cup of tea. That’s what being informal and experimental means in this context–a strong commitment to a wide range of expressive and substantive outcomes. You don’t break from that in the middle of a collaboration–if the stewards of a resource want to make changes, they do it after a given cycle of collaboration is over and they talk as explicitly as possible about their reasoning once they make those changes.

    Transparency isn’t a buzzword, it’s an ethos with as many obligations and burdens as other kinds of professional or institutional norms.

    I do think it would be interesting to invite the JDH editors to actually “take up a space” inside the post itself as well as to comment or counter-narrate (almost in the mode of a journalist seeking comments). I was curious about whether you and Roopika actually asked them point-blank, “Why are you making this change to process? Especially after we’ve met your very difficult initial deadline, with no small effort on the part of us and our collaborators?”–I was expecting that moment to come at some point in the narrative since I think a lot of us would have asked (with varying degrees of irritation or annoyance) why they were doing this.

    • Roopika Risam (@roopikarisam) August 29, 2013 at 5:04 pm #

      Timothy, I made a series of comments on Twitter about what I would have preferred the process to look like – ones that could be a useful guideline for codifying a process while preserving both a transparent and experimental ethos. It would simply be 1) telling editors they have an experimental approach to publication, 2) proposing a reasonable timeline for the experiment to play out, and 3) asking the editors if they are interested in being involved. If those were the conditions under which we had been working, a) I certainly would have said yes and b) we would easily have avoided the conflict. It’s not rocket science – it’s just being transparent and upfront.

      As for why they were making the change to the process: the change was initially presented to us without explanation. Subsequently, we spoke over video conference and tried to get an answer. What it boiled down to is that they had expected different material than we provided and they wanted it reviewed. In and of itself, that is not a problem. The problem is the way the entire process was handled – and it was handled poorly at best and with attempts at manipulation at worst. For example, we were told that they had never committed to publishing the material, which came as a surprise to us given the correspondence Adeline included here.

      Here’s the thing: conflicts between writers and editors or special section editors and journal editors happen all the time. How they are handled is a significant factor in the working relationship. In the case of JDH, they reneged on a promise and a process – not simply wanting review but not even being able to tell us whether they would use the material – and then they tried to tell us that we had misinterpreted their initial intentions, despite the fact that those intentions were in writing.

      Peer review is, at its heart, based on some degree of trust – trust in the review process, trust in our peers, trust that review will improve our work, trust in the feedback our reviewers provide. What JDH did was take our trust – and the trust of our contributors – and trample on it. Even a new model of peer review must preserve that element of trust. If it doesn’t, it’s beyond broken.

  7. Timothy Burke August 29, 2013 at 5:52 pm #

    Following up, I think it actually *is* a problem if a scholarly journal, however it is organized, solicits participation from people that it designates as having an editorial or curatorial authority. Even if the editors had been more quickly and forthrightly transparent about their change of heart, it is wrong in some sense to have such a change of heart unless there is an extraordinary reason for it. Say, that you think that the content that your guest editors are providing is very seriously racist or fraudulent or something on that scale. Otherwise you treat the invitation to collaborate as a professional commitment and wait with fascination to see what people who are not you come up with–that was the whole point of inviting them in the first place. Precisely because you really should not back out on such an invitation unless there are extraordinarily serious reasons for doing so, backing out on such an invitation MUST be read by both of you (and your contributors) as an important professional act with professional implications. And that’s where power dynamics kick in hard: whose professionalism is seen as something a person can treat lightly and whose is not? So in some sense it was never “fine” to decide after seeing material that they didn’t want it, no matter how clear they had been about it–that’s only fine with blind or unsolicited processes of submission.

    • Timothy Burke August 29, 2013 at 5:53 pm #

      Sorry quick clarification: it is a problem if they solicit material from guest editors or curators AND THEN reject material.

    • Roopika Risam (@roopikarisam) August 29, 2013 at 6:23 pm #

      To be very precise: we had approached them with material from one source (an open thread that had generated much debate), they came back to us and suggested we use material from a different source (our #dhpoco Summer School) and pull it into a special section for their September issue. Among the going stories we have been getting from the journal (and there are a few), this is the first time they were approached by editors wanting to do a section and that’s why a kerfluffle ensued. In light of that, I wonder why the experimental element wasn’t made clear to us at the outset and why the need to push us to produce the section in 4 weeks – only to turn around and then tell us not only will they not publish it now but they might not publish it ever.

  8. Michelle Moravec August 30, 2013 at 5:08 am #

    I’m so glad to see discussion occurring here, especially Natalia and Siobahn’s points.

    The decision to shift, mid-publication schedule, to a different editorial model, in a special section that questions the DH community itself for being insufficiently inattentive to identity issues, is more than unfortunate. In a world where discrimination exists, decisions must be understood within that context. It is the privilege of some groups not to understand that.

    The shift from discussion based on the agreed upon procedure for editing the special section to one of meritocracy is especially distressing in light of the long history of academics dismissing work by people of color.

    Play by our rules or go home is the message of most journals, and by and large most academics suck it up and play by the rules (Although Martha Nell Smith’s fabulous comment above reflects why those rules often don’t work]. The “rules” then need to be consistent and open, even in a journal that is new and committed to experimenting.

  9. Ernesto Priego August 30, 2013 at 9:48 am #

    Thank you for sharing your experience. There are great lessons here for everyone involved in academic publishing– from authors to reviewers, editors and publishers.

    As I expressed it to Adeline and Roopika on Twitter I am sorry they had to go through that.

    A look at the About and Submissions pages of the Journal of Digital Humanities makes it clear the editorial process they follow is not “transparent” (nowhere is the word “transparency” mentioned), and the only “peer review” system mentioned is “peer-to-peer review”.

    I think it’s important to clarify that this is not a problem arising form JDH being an “experimental” platform. If we expected transparency and openness that might come from own horizons of expectation, but there’s nowhere in the Journal’s publicly available documentation that points to that. I want to stress that the lack of transparency in the editorial process, expressed as a lack of consistency and public, open availability of any guidelines, is not necessarily a consequence of the “experimental” nature of the journal in question. It sounds more like a question of doing things the way they have often been done. (Part of a culture in which decisions are often personal and final, editors/reviewers/evaluators feel there is no need to justify their decisions, job candidates get no rationale for their rejections, etc.)

    In that sense, though it is still very disappointing JDH changed their editorial guidelines/agreement with you, it is, to me, not surprising: the combination of a post-publication “Editor’s Choice” and an opaque soliciation process (who gets soliciations and how? this is never explained– the suggestion this is also an “Editor’s Choice”). It is, no doubt, bad editorial practice, and I’m puzzled at why they have not yet given a public explanation on what happened.

    I believe we need (“we” as those academics invested in working towards new. fairer methods of doing –and getting– academic work) to develop ways of recognising innovation is never straight forward and that “experimental” platforms are likely to make mistakes along the way. It is part of the process. It is not ideal– we would like it to be a bumpless road, but it’s definitely these ‘bumps’ what also allow us to become aware of what does not work or should work better. We need platforms/authors that are willing to listen and embrace this feedback, though. We need platforms to be prepared to listen but also offer explanations of how decisions are made– having clear, public, open guidelines and policies is one way of ensuring that everyone knows what to expect when.

    I can’t wait to read a public statement from the editors of JDH where they explain what happened here. Why did they change their submission/editorial process midway with you? Why do the About and Submissions pages of the JDH fail to clearly explain how solicitations are made and how the “Editor’s Choice” is taken? Why do the About and Submissions pages fails to represent the process they subjected you to?

    In brief, I am gutted you had to go through this. On Twitter, things get amplified very quickly, for better or worse. I think the good aspects of this ‘amplification’ outweight the negative ones, but I sincerely hope this does not become an excuse to be unduly critical of experiments in academic publishing, because we really need more of them. I believe the problem you faced is not specific of the “experimental” nature of JDH. For example, wow many call for papers and ‘collected edition’ call for submissions fail to include any mention of copyright and licensing policies, or clarity regarding the editorial process? How many conference committees publicise their guidelines, and how many authors are rejected from presenting their work without any explanation/transparency about how that decision was taken?

    As your post title rightly suggests, perhaps the problem is that the lack of an open, transparent, standarised process is all too familiar. Because many have benefitted form that opacity, it has seldom been openly, publicly denounced. But the times are changing…

    Thank you for this opportunity to comment. Hopefully we’ll see a public statement from JDH, and hopefully we will sooner than later.

    • David Golumbia August 30, 2013 at 10:26 am #

      since Adeline asked me to, I’m replying to Ernesto’s post, although I largely agree with it, and what she asked me to reply to was some tweets which are a bit different.

      in part, on Twitter, Ernesto wrote, “I feel it’s important to point out this is endemic amongst traditional, accepted system” and that “anti-innovation folk [may] use this as excuse to target non-trad journals.”

      I would like to see evidence that this is endemic. In my opinion, “traditional” (a word which I believe is very unhelpful in this context as many others) journals have been among the major sites for the development of postcolonial, feminist, antiracist and many other forms of critical thought. I am sure all of them have made editorial mistakes, but I don’t know that one can say Signs, differences, Social Text, diacritics, postcolonial studies, Textual Culture, and so many others can be dismissed this way as part of an endemic, anti-POC, anti-critique culture. That seems like a really odd claim to me, given that they are publicly the face of a pro-POC culture. they are often quite open and transparent about their editorial practices. Further, my experience with several “innovative” DH journals has been much less transparent and much more subjective than with quite a few print publications.

      i don’t care about “innovation” or “traditional”–I think these are buzzwords that are used to distort the argument before beginning. All these journals were/are “innovative” and “non-traditional” in many ways, whatever those words mean. If anything, I think the pretense to having a new method that is no-method makes it easy for a “tyranny of structureless” — ie, the same old politics with a glossy new “formal” guarantee of political goodness — to take hold, as Michelle Moravec writes here: http://historyinthecity.blogspot.com/2013/08/i-already-know-what-happened-and-i.html?m=1.

  10. Jesse Stommel (@Jessifer) August 30, 2013 at 11:46 am #

    As someone that runs an experimental journal with an evolving review process, I can sympathize with JDH. This is extremely hard work. One of the problems, as I see it, is that there aren’t many role models for this kind of publishing. More than almost anyone else, though, JDH should be one of the role models.

    I guess what is especially broken about traditional notions of peer review is the way that they have become so insidious that even supposed alternatives are often nevertheless mired. It is just too easy to default to the status quo. And, unfortunately, that also means that it is too easy to default to systems of privilege, gate-keeping, and institutionalized forms of oppression.

    One of the most important points I’ve seen made in this discussion is that it doesn’t matter whether or not what JDH did was intentional. The point is that there is a systemic problem that needs to be addressed.

  11. Ernesto Priego August 30, 2013 at 1:12 pm #

    I can’t reply directly to David Golumbia’s reply to my comment so I will leave another comment here to acknowledge it. I want to clarify I never mentioned any journals (he does name journals, and it sounds like I was meaning those journals). I stand by my opinion that mainstream, ‘traditional’ academic culture is opaque. Transparency has never been a feature until recently– at least not without much fighting for it.

    On my comment I ask questions regarding conference committees and job interviews (I have stopped counting how many “unfortunately we are unable to provide feedback” automated emails I have received) that address this problem. It is a problem. Precisely, my comment wants to detach the discussion from an “experimental” versus “established” (or traditional, whatever) models. This dualism is unhelpful. Opacity is a widespread problem and because I’m on holiday and because I it’s not really my job I am not going to link to the evidence but it’s been my experience and I’m sure others will relate. Of course, there are plenty of journals and platforms of all types that follow best practices, are accountable and open and transparent. But in my opinion they are the exception– ‘blind’ peer review is still the norm, and ‘open peer review’ is still at a largely experimental phase.

    I believe ‘innovation’ is not just a buzzword. JDH are trying to do things that were not done before or that in any case they are not the norm– in this case post-publication peer review, re-publishing content that has been already published online. In this case I personally believe they made a mistake, and not prioritising it as an important issue is for me personally disappointing.

  12. michellemoravec August 30, 2013 at 1:58 pm #

    Cultural studies, gender studies, and ethnic studies all went through similar experimental phases in publishing. I find extremely interesting how few maintained the different modes. One of the few that does is Feminist Review http://www.feminist-review.com/ which clearly explains how the collective functions.

  13. Jennifer @UnchartedWorlds August 30, 2013 at 4:57 pm #

    I’m an outsider to the community you’re talking about – I’d never heard of the JDH till this post came up in my Twitter timeline. But as a writer & activist, & witness of many online arguments over the years, I just wanted to say I think you did a GREAT job of writing up this situation in a constructive way. Respect.

  14. Josh Honn September 1, 2013 at 11:04 am #

    I think the point that Michelle and David make about the tyranny of structurelessness is a very important one, and something I feel many open movements have not properly theorized or dealt with. Again, to focus this discussion back to its original intent of dealing with structures and process, as we continue to think critically and look to apply an informed praxis of transparency, especially within scholarly publishing, a paper worth reading might be Nathaniel Tkacz’s “From open source to open government: a critique of open politics” (Ephemera, 2012). I think the following quote might be sufficient enough alone to entice engagement with the increasingly reified and problematic ideology of “open”:

    “I argue that the logic of openness actually gives rise to, and is perfectly compatible with, new forms of closure … Moreover, I claim that there is something about openness, about the mobilisation of the open and its conceptual allies, that actively works against making these closures visible.”

    The article—”openly” accessible, no less—is available here: http://wrap.warwick.ac.uk/53295/

  15. Tom Scheinfeldt (@foundhistory) September 3, 2013 at 7:56 am #

    I first engaged this issue on Twitter, and that’s where my engagement has stayed — mainly because I have felt like the problems I wanted to address were mainly Twitter problems — but in fairness to its seriousness I thought I should engage here as well. Over the past week, my thoughts on the matter have become clearer, and they are basically twofold:

    1) Adeline’s criticism of JDH’s lack of transparency is legitimate. Even without all the facts (i.e. JDH’s side of the story or a more complete record of the correspondence with JDH), the fact that Adeline and Rookpika even felt blindsided by the process means that the process could have worked better. I still have some issues with their delivery of this criticism (for instance, I still don’t see why she couldn’t have waited a week for the JDH staff to return from vacation, and if you’re going to call somebody out by publishing their email, don’t cherrypick, let’s see all the email), but the criticism is valuable and even constructive. If the issue had remained there, I don’t think I would have objected.

    2) However, the critique didn’t rest there. As it moved from the blog post and its mostly well-considered comments to Twitter, it quickly became heated and escalated from claims about JDH’s lack of transparency to suggestions that this lack of transparency stemmed from some sort of race and/or gender bias. This race/gender claim hardly has its seeds in Adeline’s post (“Do our experiences indicate a–hopefully unconscious–bias within the journal?”) but the discussion on Twitter quickly took on the language of race and gender, becoming one of microagressions, etc. (e.g “Elephant in the room: a journal’s lack of transparency resulting in the appearance of different treatment for WOC = microaggression) and elsewhere in the blogosphere e.g. (e.g. “JDH had suddenly raised the bar on two junior faculty (one recently tenured, one newly hired) who “just happen to be” women of color”).

    This leap is a problem. We have seen NO EVIDENCE of any race or gender bias for JDH’s lack of transparency. I understand the structural arguments that lack of transparency will always shelter and lead to biases of these sort, and as such I agree that we should work as hard as we can to fix transparency problems before that happens. Even if it’s not part of my experience as a white man, I also appreciate and acknowledge the wariness and discomfort that many women of color and others feel when they encounter such lack of transparency, because it has so often been the source of their marginalization. But again, we have seen NO EVIDENCE of any race or gender bias IN THIS CASE. To suggest otherwise in the absence of such evidence is irresponsible.

    One related point. This dispute involves more than “structures” and “processes,” which JDH’s critics have over and over again insisted are their only targets, despite quite a bit of talk on Twitter about “naming and shaming.” Like it or not, there are real people involved on both sides. I have seen many heartfelt expressions of personal sympathy toward Adeline and her collaborators of the “I’m sorry you had to go through that ordeal” variety. Then I have seen many of those same folks turn around and say that their critiques are purely “structural.” Well. if it’s personal on one side, it’s personal on both sides. These are not simply analyses of historical events or cultural phenomena. They are critiques of the actions of real people, who will pay real prices for those actions. A career can recover from accusations of lack of transparency. And if the dispute had stayed there, well, fine. But a career, especially of a junior scholar, may not survive accusations of race or gender bias. We can be somewhat careless about the first, but not about the second. Unfortunately, in this case, I think we were careful about the first, and somewhat careless about the second.

    I am not calling for anyone to be silenced. I am calling for truth. It seems there’s some truth behind Adeline’s lack of transparency claims. Let’s talk about that, vigorously, as the bulk of Adeline’s original post, and even more so her latest post, does. At the same time, even though we have seen the leap from lack of transparency to race and gender bias made time and time again historically, let’s not jump to conclusions about bias in this case before we seen evidence of it. (Alas, I think that means, that at this stage of the game, we now really need to see the entire email correspondence between Adeline and her collaborators and JDH.)

    Anyway, that’s my $.02. I hope you’ll take it for what it is: a colleague’s well intentioned attempt to take the issue seriously. Thanks for the invigorating discussion.

    • Brittney Cooper September 3, 2013 at 9:02 am #

      There are a few problems here, Tom.

      1.) The key difference between structural bias and individual bias is the functions of intent versus impact. No claims have been made about the intent of the folks over at JDH; as you rightly point out, it is unclear whether they intend to be discriminatory, or whether the process *simply* shifted midway through, or what have you. However, the impact of their actions matter, and if the impact of the actions taken is that the review process has suddenly changed after two junior women of color submitted, then bias has occurred, despite whatever people’s intents may be.

      2.) You seem to be more bothered by the harm that claims of race and gender bias will do to the accused than what the experience of that bias means for those who have experienced it. This is a problem. It fails to acknowledge the operations of privilege. And it also engages in a problematic sleight of hand, in which the accusers merely “feel” discriminated against, while the folks who have potentially presided over a problematic and biased process have been subjected to “actual” harm. Based upon how you argue here, these WOC are merely perceiving something that has no evidentiary basis in fact. In other words, they are being hypersensitive. You said it nicely, but this is basically what you have argued. This kind of disingenuous argumentation is a serious problem often faced by people of color who attempt to point out the problematic functioning of race/gender in the academy.

      3.) You have basically set up a system in which you will never be convinced about the truth of Adeline and Roopika’s claims because even if they provide evidence, that evidence will most likely never confirm *intentional* bias. Therefore, their feelings will be left to the realm of the subjective and hypersensitive. This is precisely why folks then argue for a structural interpretation, because you have already discredited the validity of their personal interpretations as not being rooted in anything but hypersensitivity about the past. Moreover, you have again suggested that those who have not handled this well are the one’s entitled to the personal benefit of the doubt, and them only.

      4.) This is why Adeline’s follow up post outlines actual concrete steps that can be taken in these matters. But the fact that she had to outline a set of processes, rather than professionally benefiting from a set of processes that are already well-designed is just one more way that people of color are forced to do additional kinds of labor that white folks never have to think about, because white people are not structurally disadvantaged in the first place.

      And this just my .02.

Trackbacks/Pingbacks

  1. Laurie N. Taylor - August 29, 2013

    […] Adeline Koh has written an important blog post on the importance of clarity and transparency on the processes for peer review. The full posting is here: http://www.adelinekoh.org/blog/2013/08/29/journalofdigitalhumanitie/ […]

  2. Points to Remember about JDH and the Recent “Kerfluffle” | Spencer W. Roberts - August 30, 2013

    […] Adeline Koh and Roopika Risam about a special section in the journal. On Thursday, Koh posted an account of these proceedings, which began with a proposal for a “special section on the postcolonial digital humanities […]

  3. Improving the Journal of Digital Humanities | the scottbot irregular - August 30, 2013

    […] and the digital humanities blogosphere has been abuzz recently over an ill-fated special issue of the Journal of Digital Humanities (JDH) […]

  4. Weekend Reading: The Back To School Edition - ProfHacker - The Chronicle of Higher Education - August 30, 2013

    […] The Journal of Digital Humanities: Post-Publication Review or the Worst of Peer Review?  “The problems of traditional peer review are well known. Peer review is not transparent; it takes too long; the true blindness of peer review is questionable, especially in small fields; its gatekeeping function encourages the conservatism of scholarship. To address these concerns, the Journal of Digital Humanities published its first issue in Winter 2011 as an effort to work through some of these problems. JDH champions a “post-publication” model of review where its editors collate some of the best existing DH work and publish it in journal form. How well does this publication model work in practice?” […]

  5. Crowdsourcing Best Practices for Experimental Journals: Transparency | Adeline Koh - August 31, 2013

    […] Thursday, I published a controversial blog post in which I described some problems Roopika Risam (@roopikarisam) and I experienced with the Journal […]

  6. Can the Digital Humanities Be Decolonized? | Indigenous New England Literature - September 2, 2013

    […] at race, institutional power, and inequality. Koh was disgusted enough that she published a blog post about what happened. To be fair, she emailed the post to JDH before […]

  7. Against Efficiency Machines | thread & circuits - September 9, 2013

    […] for publication in The Journal of Digital Humanities. That went horribly awry, as Koh chronicles here. In the aftermath, we are moving forward with another digital humanities journal (one that also […]

  8. New Ideas; New Problems | Digital Mallory - March 10, 2014

    […] humanities are still working toward establishing conventions of peer review and publication. Adeline Koh and Roopika Risam when working with the Journal of Digital Humanities, encountered a change in procedure, and felt […]

  9. New Ideas; New Problems | #hist5702x - March 10, 2014

    […] humanities are still working toward establishing conventions of peer review and publication. Adeline Koh and Roopika Risam when working with the Journal of Digital Humanities, encountered a change in procedure, and felt […]