As promised yesterday, I’m going to write down my notes from Saturday’s final panel session of Science Online London 2010. But first, I want to say a couple of things about the Research Information Network report that Rob Procter discussed, If you build it, will they come? How researchers perceive and use web 2.0. (These concerns were raised in the twitter feed at the time, but for reasons I won’t bore you with, we didn’t have time to discuss them at the meeting itself.)
In the last half of 2008 I consulted for the Science Advisory Board (SAB) on a study looking at How Online Media Affects Traditional Publishing Methods. This study was, like the RIN one, a survey of scientists and their use of social media, or ‘Web 2.0’ (by the way, there was a tweet on the #solo10 hashtag mentioning the Britishness of saying “two point nought” rather than “web two point oh”. Was that really me?). This was an international study, rather than the British-centric RIN one, and surveyed 1500 scientists (vs 1300 in the RIN one).
What perturbed me on reading the RIN report was that the SAB study found that it was the younger and more junior scientists who were making more use of Web 2, whereas the RIN report seemed to imply that it was being driven by older scientists. In fact, one of my conclusions was that as the older guys died off we’d see more uptake. (Both studies bemoaned the low overall uptake of Web 2 tools, although the SAB was more upbeat in its assessment).
But as I sat on the Tube on the way home last night, I realized there were a couple of major flaws with the RIN study. The RIN sent its survey to 12,000 scientists in the UK, and got a 10% response rate. That’s a pretty lousy statistic. Here, we’ve selected for people who have both the time and inclination to respond to a random survey. Most of the SAB respondents were selected from the SAB’s membership (currently nearly 50,000) to receive the questionnaire, and were rewarded for their participation (the SAB operates a points system: if you respond to questionnaires and whatnot you can accumulate points which can be exchanged for physical goodies). The response rate was a lot higher (I don’t have the exact numbers to hand) and we might assume that the quality of response was correspondingly higher, too.
A more worrying question, however, is how were those 12,000 people (who received the RIN survey) selected in the first place? Turns out that these are ‘scraped’ email addresses, which makes me think there was already some bias towards older, more well-established scientists in the first place. Young researchers not only have had less time for their contact details to be established on an institutional website (and indeed, pre-tenure, have probably moved around a lot, relatively speaking. Google me, for example; the second hit is me but the email address in it was defunct five years ago) but are possibly also more security conscious and less willing to have their email address available for scraping.
I think those two concerns might well go some way to explaining the ‘surprising’ results from the RIN study.
So, what did I say at SOLo 10, after all that? Here’s my notes (like Ed Yong, I have to refer to my Moleskine notebook. No copy & paste here, and it’s not a transcript!):
< fx: English accent, ginger hair>
This is a room full of very special people <fx: laughter>. If we didn’t believe in online technology, the value of it and the coolness of it, we wouldn’t be here today.
And over the last couple of days we have seen some very neat stuff. This morning, Aleks talked about the Growing Knowledge project; Peter Murray-Rust showed us a really cool experiment this morning, and we’ve had a whole heap of open- and linked- data stuff–semantic web, if you like. Real nerdgasm stuff.
But, we have to remember, we are special. We are the early adopters, if you like. To borrow a phrase from technology business development, we haven’t yet crossed the chasm to mainstream adoption of these cool toys, as Rob has just pointed out.
The vast majority of jobbing scientists simply haven’t signed up yet, perhaps for the reasons Rob listed. We, here in this room, are a load of technology evangelists, there are a few companies here who share that vision and who have demonstrated some of their toys, but people as a whole?
No.
Of course, they’re into Web 1–email and websites and whatnot, but Web 2, Web 3? Not so much.
We’ve given reasons over the last two days why people should adopt these technologies, but there’s been a lot of stick, and not enough carrot, I feel. What should we do? Encourage–or bully–people into using this stuff, just because it’s there, just because it’s cool?
I don’t think so.
I think, rather, that it comes down to two things, and my thesis is very simple. People, as a whole, will only adopt these new technologies for one of two reasons.
First, these new tools allow you to do something necessary, something you have to do anyway, something that exists outwith cyberspace but that you have to do, but that is made so much easier, so much more efficient with internet tools that people will WANT to do it.
Obvious examples are PubMed–anybody remember Silver Platter?–and online journals themselves. When did you last use a photocopier to copy a journal article?
Say what you like about PDFs, didn’t life get a lot lot easier in the late ’90s and early 2000s?
The second thing that works is something that adds value, and that value can include ‘fun’, but a value that just can’t be gained from anywhere else. A compelling value. For example, Facebook and Twitter are great Web 2 tools that allow people to communicate in new and exciting ways.
It’s slightly less Web 2, Web 1.5 perhaps, but Faculty of 1000 I think is such a tool. We’re addressing the filtering, the information overload problem, adding value to the published research. We don’t care, actually, whether it’s Open Access or where it’s coming from: we’re just providing editorial, if you like, content on top of the literature. And you can’t do that, effectively, without cyberspace. It won’t work.
The challenge, really, is not to have a smart idea. There must be oh, how many people are here? 120 bright ideas in this room alone. But you have to figure out where that value is, that compelling calue that will make the vast majority of scientists want to use this stuff we’ve been talking about.
This includes things like blogging networks, like data visualization, like linked datasets.
and then we were out of time. There was some ad-libbing in there, but that’s the gist. Oh, and I had no slides.
(If you would like to comment without signing your soul away to Nature, this post is mirrored at the BioLOG).
Weren’t there demographic questions asked in the survey? This would reveal any odd skew. Agree the findings are odd.
Nature (italic) is the journal. Nature Network is an internet platform (which is also the internet platform which hosts Nature journal content).
Irrespective of one’s views on registering in order to comment, Nature knows nothing whatsoever about anyone who comments at Nature Network. (They wrote up this report some time back, also).
In purely marketing terms, 10 per cent is a good rate of return. I believe that 0.3 per cent is considered good in a standard marketing survey. Of course you’d expect to get a higher return from a list that is pre-filtered, as it seems the RIN list was, from what you write.
Most surveys are flawed for many of the reasons you describe, and others. I personally would not take any notice of a survey without being able to access the raw data, to be able to look for myself at the demographics, response rates, actual questions asked, etc. I don’t know if this particular survey comes with a supplementary information file of these data. I recall receiving it as a brochure, but not whether the actual data were included in addition to the summary.
Last page of report has the age breakdown:
under 25: 64
25-34: 398
35-44: 385
45-54: 325
55-64: 233
65+: 60
2% of the under 25 group were bloggers. Here is some math:
1/64 = 1.6%
2/64 = 3.1%
So there’s one student blogger in the entire sample. (Maybe an older grad student or two in the 25-34 group as well) Hm.
bq. Nature (italic) is the journal. Nature Network is an internet platform (which is also the internet platform which hosts Nature journal content).
Funny you should say that, because when you’re on this page and signed in, clicking the ‘Account’ button at the top right takes [me] to [my] Nature details, not the Nature Network ones, as I’d have expected. That’s irrelevant to the point anyway; which is that people are put off commenting because they have to open an account.
0.3% might be good in marketing, but this wasn’t marketing, it was research.
Thanks for doing the nerding, Eva. What percentage of all sci-types in the UK are bloggers, I wonder?
I agree it is surprising if web 2.0 activity is being driven by older scientists.
When I first started using the net in the early 1990s, I recall that much of the discussion in Usenet groups about science seemed to be driven by postdocs/postgrads rather than senior people. It seems unlikely that this would be reversed today, though I would expect some levelling out.
I was thinking about this on the way in this morning, and it struck me that I, too, remember Usenet, as a postgrad, and that means that, um, I am starting into that ‘older scientist’ bracket.
That can’t be right, shirley?
Well, you are in the age bracket that the RIN report found the most active online, so maybe there is a “Usenet-effect” there. I didn’t get online until 1996, and never really got into it. It was all becoming web-based by then.
I’m afraid middle age claims us all eventually, Richard… welcome to the club.
PS And don’t call me Shirley…
Nonsense Austin. I’m going to live forever.
Heh. Believing that is, of course, one definition of youth! Good trick if you can manage it…
…or were you thinking as an online cyber-entity?
Yes. But if we can’t live for ever, we’ll have to make do with multiple copies.
But it’s a serious point here–those of us who were very active ten years ago and are still active are weighting the distribution towards middle age. The question remains, are the younger ones as involved with Web 2 as we were with the internet then? The RIN study doesn’t really have good data on this, for the reasons above.
I suspect that free computer time is being sucked up by Facebook et al. these days and that yoofs mostly view out-of-hours computer use as an opportunity for social or leisure stuff. You have to care a lot about x to use that spare time blogging when you could be commenting on your mates photos and playing (*ach*) farmville. Leisure = fun; social = all sorts of vital human stuff that predates even language; working during playtime = ..?
Hello Chris, thanks for dropping by.
Yes, you might be right. Hence my comments about compelling value.
@Chris, yes good point. But it’s ironic that some non-scientist people are using their free time on scientific pursuits like Galaxy Zoo and FoldIt.
As I pointed out, ‘fun’ might be a subset of (or at least overlap) ‘value’.
I append Rob Procter’s comments (from my ‘open‘ blog) :
On your first point, 10% is actually quite a respectable response rate for an email survey. The important issue, however, is that reliability of survey results cannot be judged on response rates alone. The representativeness of the sample is critical, which brings me to your second point. As the RIN report documents, the profile of respondents as defined by age, position, discipline and gender shows no significant bias when compared with that of the profile of the UK academic community as a whole (including post-graduates) as determined from Higher Education Statistics Agency data. So, there is no evidence to support the suspicion that there was a bias towards older, more well-established scientists among our respondents. Of course, there may be a selection bias, such that those who chose to respond might be more (or less supportive) of using Web 2.0. David Crotty on his scholarly kitchen blog (http://scholarlykitchen.sspnet.org/) has suggested that ‘luddites’ might be underrepresented, such that our survey results actually exaggerate the extent of Web 2.0 use.
The results of your SAB survey are interesting, but I would like to know is how does the membership compare with the academic research community as a whole? The number of respondents is impressive but you will have to explain why you think you can assume the quality of the response was correspondingly higher.
The differences between the findings of the two surveys regarding the engagement of junior researchers are interesting and I will certainly take a close look at you report. It may be that there are significant national differences (about half the SAB membership seems to be drawn from North America). What I can say is that our findings are in line with those of a recent JISC study (http://www.jisc.ac.uk/news/stories/2009/11/generation.aspx). The first annual report concludes (p 57):
“This research has not supported this assumption (i.e. that Generation Y would be early adopters and keen users of the latest technology applications and tools in their research). On the contrary, it would appear that Generation Y doctoral students, in common with others, are quite risk averse and ‘behind the curve’ in using digital technology, not at the forefront; and this despite the fact that the majority of Generation Y students answering the survey and in the cohort appear to be keen users of the latest technology applications in their personal lives.”
There were a total of 15 postgraduates in our survey who claimed to frequent bloggers (i.e. at least once a week).
I’m curious to know – where don’t you have to sign in to comment? I have accounts at Typepad, Blogger (Google), Facebook, Twitter, etc, and I have to sign in to all of those to use them. I know I have asked before (not on this blog) about this reluctance to sign in and I still don’t fully understand what negative thing people think will happen if they do, but is there a platform that does not request people to sign in (and isn’t full of spam)? Thanks!
Maxine, I’m talking about creating accounts. That is not the same as signing in. You do not have to create an account at any of the other blogs that I administer: you simply enter your name, your email address and your website if you want to. You can check the box that says ‘remember me’. But there is no password, no double opt-in (unless you want email notification of follow-ups); there is also very little spam.
The system here raises a much higher barrier to making a casual comment, for what appears to be little benefit. (NN is not a social hub like Twitter and Facebook–it’s far too clunky for that.) We know, too, that demographic data have monetary value.
As you are a Nature employee, I am beginning to find your continued defence of the NN system somewhat tiresome and counter-productive. You have a vested interest in the success of [Nature]–as a publisher, social media hub, whatever. Fair warning: this platform does give blog owners the ability to ban commenters, and I will make use of this if anybody persists in what I consider to be destructive behaviour.
Wow, Richard, please don’t worry. If you feel that way I will refrain from commenting at your blog. I had no idea that asking a question in response to a comment from you would elict a response in such aggressive terms. Happy to end this “conversation” here!
Now that’s sorted out, maybe we can return to the subject.
The RIN study includes non-scientists, as well as computer science & maths. They, I suspect, will be under-represented in the SAB (the SAB being geared more towards biology and medicine). In fact, Comp & Maths are disproportionally frequent users of SM, and maybe it’d be interesting to remove them and re-analyse?
The keenest users (RIN) by position appear to be Research Assistants, so I’d like to know the age-breakdown there: it’s pretty flat across the other positions. I’d also like to know the age break-down by discipline (humanities-types stereotypes and whatnot).
There’s also a confusing line in the report, where (page 6) it claims that "junior and younger researchers [are] more likely to be frequent users of social networking." That goes against the 35-44 age-group stat, but I can’t find the data for that.