So we’re launching an ad network…

Why? Technorati was founded to help bloggers succeed and to bring audiences to blog content. Given our unique position of running a blog search engine, an ad network geared towards helping blog and social media publishers at every level to make some money just made sense.

We’ve been successful attracting premium brand advertisers to Technorati.com – and we’d like to extend those relationships for bloggers, as well as give our advertisers the deeper reach into blogging and social media they’ve been asking for.

Our first step was a private beta. We assembled a core of like-minded sites, founded to provide community and services to bloggers and to surface the best of blog content to consumers, and were successful in attracting advertisers to the network including: T-Mobile, Toyota, and Verizon.

These sites form the base of the Technorati network’s vertical content channels and reach an audience of 17 million (with that audience increasing very shortly with several other sites about to sign). Over the next several months, we’ll be adding blogs from the mid and long tail within those verticals. Here’s some of who’s in so far:

• blogtalkradio
• Blogcritics
• blogcatalog
• BlogTV
• GeekAlerts
• GPSMagazine
• NerdApproved
• Technabob

That doesn’t mean we’re moving away from our core. We’ve organized the company into two operating groups – the network and Technorati.com. Blog search is still and will always be the foundation of everything we do.

In our biggest internal initiative, we’re in the midst of a summer-long project to completely rewrite our crawler and search engine. Last week, Dorion addressed some of our recent challenges and fixes. An updated search infrastructure should address of the vast majority of the complaints we receive, greatly reduce spam and give everyone a faster, more efficient utility. You’ll also see significant upgrades to blog claiming and Technorati Authority. Our product team has also spent a lot of this year getting feedback directly from the blogging community and incorporated this into the development of our widgets – as we roll them out a lot of you will recognize what you see.

You’ll see some new features designed for our readers as well, but I’ll leave this for a future update.

Well, I'm disappointed that I have to tell you what your probably already know, we have stumbled a bit the past two weeks. Page load times have been on the rise over the last week and, in a couple instances, the site has been nearly unusable. We are working exremely hard to resolve the underlying problems, but I thought it important to let you all know that we are keenly aware of the problem and, like you, want the site to be screaming fast.

What happened?

A high volume of automated processes (AKA "bots") access our service for various purposes, many with nefarious ones, exhibiting onerous behaviors and often masquerading as human users. These bots can, at times, impact the stability and performance of the service.

What are we doing about it?

We have an ongoing effort to reduce the impact of bots. At times we've had to throttle certain activities particularly around feed and API requests. We continue to upgrade and add new hardware as load dictates. We are also configuring new detection and prevention mechanisms to help ensure that real end user requests are our top priority to serve.

When will this all be done?

Several defenses have already gone live over the past week and these additions have resulted in a significant reduction in backend resource consumption and have stabilized parts of the overall system.

We constantly monitor the system and, as of this writing, have been able to cool things down again very close to our desired response time levels.

We appreciate your patience and understand that many of you have come to rely on our services in your daily use of the Internet. As I stated before, we strive to provide a great user experience. I want to thank the dedicated team I work with who is getting us through this difficult time. I hope you too can thank them when we achieve our goal once again.

UPDATE:

Well, we've had over 15 hours of excellent response times from the system. We have addressed the underlying capacity shortage things have returned to normal. Thank you again for your patience.

According to Yahoo, 51% of internet users will turn to blogs to gather information and communicate about politics. Citizen journalists are the ones posting the stories that break through the campaigning and ask the hard questions.

Authenticity is what plays with this audience. Spread misinformation or spin, and more than 30,000 political blogs (tagged politics in the Technorati index) are ready to call foul.

There's a brilliant application of Technorati data over at Tech President. (Disclosure: co-founder Micah Sifry's brother David Sifry is Technorati's founder.)

View Technorati election data profiles.

Taking a pulse of the blogosphere today, what do the numbers tell us? (Keep in mind that Technorati is indexing in real time, so the numbers can vary even by a few minutes.)

Barak Obama has pulled into the lead – in terms of attention in the blogosphere. If 2008 is truly the social media election, as has been posited, all signs point to yes. As the only Republican candidate, should John McCain be benefitting with a focus in attention – or will he rebound once the Democrats have picked a candidate?

Simple and telling: the tag cloud on Technorati's politics page.

Hilary Clinton

English posts that contain Hillary Clinton per day for the last 30 days.

Get your own chart!

Barak Obama

English posts that contain Obama per day for the last 30 days.

Get your own chart!

John McCain

English posts that contain McCain per day for the last 30 days.

Get your own chart!

So you’ve installed WordPress 2.5, now you want to show Technorati links on the dashboard. Here’s the code...


read the rest

We're seeing thousands of blogs per day that we're not indexing because they're bearing symptoms of being compromised (see the previous post on the matter). If you're not using versions 2.3.3 or 2.5, you must upgrade to protect yourself (perhaps 2.0.11 and 2.1.3 each fixes this issue too, I'm looking for confirmation on that).

Blogs that have been compromised by this security vulnerability are typified by having links to spam destinations inserted onto the blog page. These link insertions may be invisible to casual observations; the links are often obscured by style attributes that render them invisible. These links are still seen by crawlers such as Technorati's, Google's and Yahoo's. You can find these links by viewing the source of the blog pages or, when using Firefox, looking under "Tools" -> "Page Info" -> "Links". Blogs hosted on wordpress.com are not affected by this issue; only blogs hosted on their own installations of WordPress from wordpress.org require concern.

Because of this ongoing problem, we're discontinuing processing crawls of blogs that exhibit common symptoms of being compromised. We strongly recommend upgrading your WordPress installation. Even if you haven't been afflicted by a compromise, by the time you are aware that you have been a number of negative consequences may have already occurred (for instance, flagged spam by Technorati, Google or Yahoo!) -- this has been reported by many WordPress users.

If you have questions about installing WordPress or maintaining a WordPress installation, please refer to the WordPress Documentation or the WordPress Forums. If you feel that your blog is not vulnerable to this hack but your WordPress blog is not being updated, please contact Technorati support staff.

Technorati's spiders were shutdown for several hours on Thursday and various intervals since then while we investigated a number of anomalies that were appearing in our data; essentially, a small percentage of recently created blogs were having their data scrambled. An example of this appears in this blog post. The spidering outages allowed us time to investigate, diagnose and make corrections that prevented further data corruption. We started running some corrective measures on Friday but found over the weekend that that was only partially effective. Technorati handles a large volume of data everyday; isolating and devising remedies for these kinds of issues that effect a small percentage of the data flow is tricky. However, we think we're recovering now and the backlog of data processing is getting worked through.

Just to peek into the works a little bit, many distributed data systems rely on centrally dispensing identifiers for data elements and Technorati has such a beast. What was found were cases of blogs new to our system (from within the last 3 weeks) losing thier identifiers and those identifiers getting re-associated to other new blogs. No blogs that existed in our system before Dec. 18th (the vast majority) were impacted at all. The outward manifestations visible were posts for blogs with a shared ID mingled (a mashup the authors naturally were unhappy with) and mis-associated blog claims ("And you may tell yourself, this is not my beautiful blog").

This was a unprecedented case for us; while it had been occurring in about 8% of those blogs (created on or after December 18) for about 2 days (beginning on Tuesday, January 8th) we had until that time never encountered this phenomenon. An intensive investigation was launched, reconstructing operational timelines and correlating facts. What we found was that this stemmed from a failure incident with the primary system for identifier dispensing, another failure in the secondary system that took its place and then a corrupted data set mistakenly taking over that one, ouch! The first two blows appeared to be handled routinely but the third time was cursed; propagation of corrupted data was not detected for about 48 hours between Tuesday when it started and Thursday when we pulled the emergency brakes on the spiders.

So we're recovering now, most of the data is being restored to its previous state and we have had a number of internal postmortem discussions about earlier fault detection and recovery. If your blog was created in our system within the prior three weeks (since December 18th) and you're seeing aberrant data associated with it or it's no longer there (try http://technorati.com/blogs/YOUR_BLOG_URL to check), please visit the support request page. A selection for 'The January 8th System Outage' will be available this month while we shake out any remaining issues that aren't covered by the remedial action under way now.

Examples:

http://chinese.engadget.com
http://desktops.engadget.com
http://hdtv.engadget.com
http://storage.engadget.com

Well, we fixed the bug yesterday. The impact of this change is mostly limited to the Top 100 and the overwhelming majority of the blogosphere is unaffected. Thanks for bearing with us while the Top 100 experiences some turbulence.

We're always thinking about how to improve and develop new meaningful metrics for the blogosphere and we welcome your feedback on these issues.

The significance of this is that the Ping-O-Matic XML-RPC interface is the default ping destination used by Wordpress installations, as well as some other blog CMS'. If that's the case for your blog and it was not crawled this week to pick up a posting you've made, please ping us directly ("When in doubt, ping the direct route!").

Here's a tip: when you see the link to ping your claimed blogs on the Technorati home page or on the ping page itself, drag that link to your browser's bookmarks and put it on the browser toolbar. Then, whenever you post to your blog you can conveniently hit that bookmark. That ping will come in to us directly and, as long as you're logged into Technorati, be given high priority in our crawl queue.

Since that time AOL, Wordpress, Vox and other great services have released their support for OpenID as well. Yesterday, I was very excited to see Eric Case post that Blogger is now supporting OpenID commenting! The more we can reduce the password-overload and identity fragmentation with all of these services we use, the better. I thought it might be helpful to show you how you can use your Technorati profile to authenticate for commenting on Blogger blogs.

First, you must be logged into Technorati to begin with. When you're reading a Blogger blog and want to submit a comment, look where the form says "Sign-in using" and select "Any OpenID".

The URL box that opens up just needs the URL for your Technorati profile, which has the form http://technorati.com/people/technorati/USERNAME. Put that URL in the "URL" box and hit the "Publish Your Comment" button.

The final step is to tell Technorati to permit Blogger to know that you're logged in to Technorati. You can grant that permission for the future so that Blogger can always get that confirmation from Technorati or make it a one-time access. Click "Set Permission" and you're done!

There are a lot of great resources and services available for learning about OpenID. I'm expecting 2008 to finally usher in the time when this stuff becomes more manageable. If there are other identity services that you would value as a Technorati user, please let us know!

Search & Discover

Our search application surfaces keyword, tag and link query results as they unfold. Our discovery features will tell you what's going on, not requiring that you know what to ask. In the 1.5M blog posts passing through our turnstiles everyday, some of them have to be good and we want to help you find them.

The Technorati Percolator combs the sea of posts and other media flowing through our systems to find the ones that are emerging as significant at any given time. Finding the needles in a fast-moving haystack and organizing them into topical groupings isn't easy. Items in the Percolator are sampled from our update stream, primarily ranked by the age of the item, the authority of its source, the authority of the referring blogs and the density of recent links to it. We found that by taking all of these factors into account, an effective algorithmic filter and magnifier emerges. A lot of great applications have already appeared on the landscape that try to solve this kind of problem. From what we can tell, those applications started with a small corpus of blogs and grew their coverage from there. Technorati has come at the problem from the perspective of starting with broad coverage, sampling it and winnowing it down to the good conversations. Of course, if you want to explore the social connectivity, Technorati's search systems are there to help.

Our primary goal with the Percolator is to highlight the significant things grabbing the blogosphere's attention regardless of the blogger's "A-list" or "Z-list" status. Our broader coverage should enable us to better serve the broader blogosphere. Yes, we have stories and sources from the main stream press as well as the "A list" bloggers we're all familiar with but we're also striving to provide more comprehensive coverage by going deeper into our data set than "page one". By exploiting our broader coverage, we're seeking to move meme-emergence applications further along the long tail.

While we're very proud of this release, the Percolator is a work in progress, so please keep your feedback coming. We're going to continue iterating on our technologies to better serve the blogosphere and help those navigating it search and discover what's happening.

Finding the good stuff in a world of 100 million sources

But the incredible growth in user generated content means we need to go beyond the "search" paradigm and find a better way to highlight and present the best of what's happening now, and moreover what's gaining attention in topical areas of interest to our audience. It’s a rich problem to solve. We're very good at showing all the latest posts on a topic, but the latest is different than posts that are actually rising in attention. It takes a few hours to a couple of days for an interesting post way down the long tail to build a string of inbound likes and start rising in attention. By which time its probably lost on Technorati search result page seven.

I suppose we could build a system that looks at all the posts on a topic from a tightly proscribed white list (there are many services that do this) but that loses much of the emergent serendipity of the blogoshere and excludes everyone except for a proscribed static elite. That’s neither fair nor interesting nor scalable. We're interested in what the whole blogosphere has to say on a topic. But to find the interesting relevant stuff we want to give a little more weight to bloggers that are revealed to be authoritative in a subject (not just because they say so in a tag, but because we observe many other topical bloggers linking to them in a democratic vote of editorial goodness).

Social, meet mainstream media

But bloggers don’t just link to other blog posts; one of the most powerful things bloggers do is collectively vote on the relevance of mainstream media articles at any moment based on linking behavior. That too is a key measure of the blogosphere's attention. It's also a key measure of a news article's relevance, because blog links are essentially editorial selection. And since blogs and mainstream media are, via links, reciprocal participants in a global conversation, we knew we had to put both on the same page to do each justice.

So, we incorporated all this thinking (and more) into designing the new Technorati.com.

We have harnessed our best-on-the-Web blog index and search to power a rich, topical experience that’s complimented by new and improved search. And this is just a step, one of many more to come towards leveraging our blog search heritage to provide richer, more relevant, algorithmically-driven search and discovery that connects our Web 2.0 startup roots with our blossoming business.

But this relaunch isn’t just about new pages. In parallel to public facing events like today’s launch, we’re also working hard behind the curtains to build our infrastructure and business, with initiatives like improved search/site performance, and streamlined monetization. This release is momentous, but it’s also multi-faceted, and to do the nuances justice I’ve asked a couple veteran team members to post some rationale and descriptions of the new features. We’ll be listening for your feedback as we progress from this step forward, and we hope you enjoy the new discovery.

Forty-eight hours, a lot of links and a fist full of poker chips later, Blog World Expo 2007 comes to a successful close. I'd like to congratulate Rick Calvert and the entire Blog World crew for putting on a very impressive inaugural event that brought together intrepid bloggers and blog businesses. Technorati was happy to support the cause, and it was great meeting the core users who make our service so relevant.

A few highlights:
-watching our new CEO Richard Jalichandra make his public debut on the Featured Keynote Panel
-schmoozing with bloggers and businesses at The Conversation Group's mixer in the Wynn
-sponsoring the opening night party at The Hard Rock Hotel
-playing poker w/ fellow Expo'ers at the b5media poker table
-losing most of my chips to savvy HireAHelper.com CEO Mike Glanz... I think I'll use his site to hire a Hold 'Em tutor for next year

UPDATE: CEO Richard Jalichandra was interviewed by WebProNews after his keynote panel:

I’ll certainly be there, and so will Technorati. We’ll have a booth both days of the Expo and we hope you stop by. There'll be a lounge so you can hang out with fellow bloggers, and computers so you can blog live, create a Technorati account, or rock your Technorati ranking for all to see. And you definitely won't want to miss this: our new CEO Richard Jalichandra will be kicking it on the couch Thursday from 1:00 to 2:00 PM. He specifically requested you stop by.

Mike Arrington and Om Malik are coming. I hear Mark Cuban will be debuting a new dance number. And Technorati's new CEO will make his first public appearance on Thursday's featured keynote panel. We, along with a cadre of other awesome companies will be there, and we hope you will too.