News tag quality

I’ve conducted an interesting experiment in social media news quality over the past few weeks. I’ve been calculating the density of tagged articles on “social media” or “social networking” in Google news alerts and in bookmarks listed on delicious and digg. Now of course there are problems with the research. Everything is based on the quantities as expressed by Google and delicious and digg. This makes the experiment results somewhat problematic, but the exercise is still useful as a launchpad for more sustained research on tag quality and the value of tagged feeds for either commercial or research purposes on an ongoing basis.

Background

During late July and throughout August I had observed a drop in the usefulness of my Google news article feeds for my tags “social media” and “social networking”.  Yet I noticed that Digg, Delicious, friendfeed, twitter and specific URL sources (Mashable, ReadWriteWeb, etc) were still consistently valuable resources for quality articles on the same tags. I theorised that mainstream media was at fault for lowering the quality of my tagged feeds, as the specialist sites and human-filtered mechanisms for tagging were still delivering quality results.

Thus my hypothesis was that as a tag grows in mainstream media acceptance, then the quality of posts and articles covering the subject area – the news tag quality – declines. I roughly defined quality for my purposes in terms of elimination criteria rather than inclusion criteria. So I regarded as poor quality links and news articles which were:
– overt or thinly veiled press releases for products;
– about statistical research reports commissioned by a company that stands to gain from the results of the research;
– lists of things businesses/individuals could do to ‘kick start’ their social media practises;
– stories about social media supposedly encouring copyright infringement, privacy invasion, job loss or some other activity designed to instill fear in users;
– repetitive, fail to report anything new, or make an obvious statement about the history of the internet.

Anything else was listed as either fair quality or quality. I collected results over a 6 month period from March to August 2009, and noted the change in the numbers of articles over that period.

Results: Tag Quantities

Quantitative results for tag searches

Quantitative results for tag searches

The google news results seem unrealistic, but it is possible that mainstream media has suddenly massively increased the number of articles tagged “social media” and “social networking” over the period.  And whilst I wasn’t able to search by date in Delicious, there was an opportunity to look at a rough graph of tagged items over the period (see below). Interestingly, “social media” tagged posts seem to be massively increasing in August on Delicious, whilst those tagged “social networking” seem to be reducing.

Articles tagged with "social media" on Delicious

Articles tagged with "social media" on Delicious

Articles tagged with "social networking" on Delicious

Articles tagged with "social networking" on Delicious

Results: Tag Quality

Once I filtered articles for my definitions of quality, the results were consistent to me, but their relevance is questionable. It is not necessarily useful to list full results, but I’ve included below my perception of the frequency of quality links per tagged feed.

Perceived percentage of quality tagged items per feed

Perceived percentage of quality tagged items per feed

The percentages were based on my perceptions of the number of quality items I received per tagged feed over the course of a month, and is based on click-throughs to articles on my browser history for the period.  A more thorough piece of research would be required to accurately determine the quality of items produced, and should be conducted based on a browser history and time spent on pages to determine the perceived quality of article content.

Research Note

I don’t want to suggest that quality articles are not being produced; rather, that among the quantity of mainstream news articles being released on “social media” and “social networking” that in generic news aggregation services, much of what could be regarded as quality reading is being lost to the masses of poor quality articles being produced.

The advantage of this personal experiment is that it reinforces the notion that feed sources used for quality information should increasingly be human-filtered (ie: use friendfeed and twitter groups to source quality articles) and any automated news feeds on specific tags should probably be updated in accordance with the quantity of articles being produced for any tag. Once the quantity of articles for a specific tag search rises dramatically over a short period, users need to refine their tags to ensure that quality is sustained.

Be Sociable, Share!
This entry was posted in Uncategorized. Bookmark the permalink.