Google trouble: Blogger and Search

Google’s getting a lot of flack for a less than spectacular launch of the gPhone (Nexus One). I’m not too worried about that, I think they’ll get it working. I’m also still optimistic about the Chrome OS netbook – though if it costs over $200 I’ve got yet another public mea culpa waiting.

On the other tentacle, I am getting bad feelings about two Google services I have long relied on – Search and Blogger. I think the problems may be related.

Blogger is the proverbial coal mine canary. It is clearly not thriving. There’s still no iPhone or DROID app for posting or editing, there’s an undocumented and unfixed 5000 post limit, there’s no mobile-optimized version of blog pages, the BlogThis bookmarklet was never updated to support categories, the rich text editor has many longstanding bugs, there’s no spam detection on comments, the Blogger in Draft blog was silent from Nov 28 through Jan 20 (yesterday!) and so on.

Why isn’t Google investing in Blogger? My best guess would be some mix of

• Inability to manage Blogger spam blogs (splogs)
• High success rate of search index poisoning comment spam
• High rate of click fraud related to Blog associated adwords
• Low rate of revenue from Blogger adwords
• Declining readership numbers
• Failure of the confusing “Follower” and Google Reader note/comment programs
• Confusion from the rise of Twitter (confuses me too) and Facebook

Several items on my speculative list implicate search index poisoning problems. These “Search engine optimization” scams degrade search results, which leads to a spiral of click fraud and declining ad word revenue.

Which brings me to the bigger Google problem. The quality of the search results is deteriorating. On technical topics that I search on, I’m getting a large number of junk web sites. I have to use my Google custom searches to find good results. When I search on hard-to-find answers that I know I’ve addressed in my own ad-free tech blog, I don’t get any useful hits at all. It’s not just that I don’t find my marvelous stuff – I don’t find any answers anywhere.

In several instances, Bing has done better. In particular, Bing seems to find fewer splogs and fraudulent ad-heavy pages – perhaps because the scummy SEO gang is still optimizing for Google. (Bing’s time will come.)

Google is only as good as their search engine, and that engine is under relentless attack from the same emergent attacks that killed usenet and severely wounded email. At the moment, the parasites are winning – and threatening to kill their current host.

Google needs a winning response. They’ve got bigger problems than lousy phone service.

Update 1/21/10: See comments for a response from one of Bloggers Product Managers, it's an encouraging rebuttal. Per that comment I corrected the name of the Blogger in Draft blog; the official Blogger blog is http://buzz.blogger.com.

Is there a club for people who hate OS X permissions?

I'm looking for a club made up of people who hate Apple's brain-dead OS X permissions/security scheme.

In the latest installment of OS X misery consider a file on a shared 10.5 drive. Whenever I edit the file from a 10.6 machine it's saved in such a way that my wife loses edit permissions -- even though both she and I have read/write permissions on the parent folder.

OS X needs to abandon its broken unix-style permissions and imitate Windows 7/Vista/XP/2000/NT. (The admin/user issues with Vista to NT weren't related to the permissions model - but that's another post.)

Grrrr. I wish the OS X customer base were way more demanding. Insufficiently demanding customers are one of the three banes of modern commerce (Two others: lock-in and fraud/deception).

See also:

• Permissions bug in OS X 10.5 - unable to update applications
• Gordon's Tech: Permissions: the most messed up part of OS X
• Gordon's Tech: OS X Leopard 10.5: The unknown user and group bugs

Update 1/18/10: No sooner do I write this rant that I have to figure out how to fix a novel permissions hassle related to moving a VMWare Package between users. This stuff is seriously evil.

Update 1/19/09: See comments. Inspired by Andrew W, I dredged up a memory of John Sicracus's famous 10.4 review telling us that Apple was going to fix their broken permissions model years ago! Today in their OS X server marketing you can read (emphases mine) ...

Mac OS X Server supports both traditional UNIX file permissions and access control lists, giving administrators an unprecedented level of control over file and folder permissions. With access control lists, any file object can be assigned multiple users and groups, including groups within groups. Each file object can also be assigned to allow and deny permissions, as well as assign a granular set of permissions for administrative control, read, write, and delete operations. Mac OS X Server supports a file permission inheritance model, ensuring that user permissions are inherited when files are moved to the server and rewritten when files are copied to the server.

ACLs have been used in the Windows world since NT inherited them from OpenVMS. This is one of several areas in which Windows has been far ahead of OS X.

The problem, of course, is that Apple has not provided an equivalent of Tiger's Workgroup Manager GUI in 10.6 standard to work with ACls, and they presumably break a lot of current software. Apple gave up on the 10.6 migration to ACLs, perhaps because of the Intel migration and the introduction of the iPhone OS.

Sandbox provided an ACL control GUI for 10.4 10.5 users, but it's not been updated for 10.6. Apple does allow us to download their Server Admin Tools which can reputedly edit ACLs on non-servers. (It only installs on OS X server.)

See also:

• The ACL Permissions pane: great review, part of the TinkerTools utility.
• Make Mac Work: Control File Access With ACLs

I'll have to continue this one in my tech blog. (BTW, Bing did better than Google at finding these references.)

Update 3/11/2010: Why you shouldn't use OS X ACLs.

Update 3/11/2010b: I try to write to a network share. I run into the 10.6 MobileMe cannot log in as other user bug. Then nothing seems to happen. I have to kill the Finder. On the other machine I discover over 45,000 0 byte files have been written. Permissions bug. I despair.

Update 8/2/10: Take Control book on permissions in snow leopard is out.

--
My Google Reader Shared items (feed)

Why we need Google Book scanning - the End of Eternity

At a small but classic library in West St Paul (which is south of St Paul, but on the "west" side of the Mississippi) I came across a book from my childhood: The End of Eternity by Isaac Asimov (1955). It's a time travel book, full of cliches.

Except they weren't cliches then.

At the above Amazon link you'll find "We don't know when or if this item will be back in stock." That's sad. The End of Eternity is not a classic book, but it's a fun book by a man who wrote a lot, and got good at it.

Over at Google Books we learn that the End of Eternity was digitized Mar 25, 2008 at the University of Michigan. We can't read it though. Under current US copyright law it goes into the public domain at about the end of eternity. (You didn't realize copyright was now effectively eternal? Missed that one eh?)

Google gets a lot of flack for their book project. I'm sure they're imperfect, but I think they're fundamentally right.

Go Google.

Update 1/20/10: Ok, so I could have picked a better example. Charlie Stross tells me I should have looked a bit longer (52 reviews, 5 stars). It seemed like such a good example at the time! In my defense the reviews are quite old, and refer to the book as "hard to find" in 2000.

Update 1/20/10b: Charlie wrote this long post today. Google is not his friend. Mea culpa.

Who killed Instant Messaging?

I know the smell of a dying solution, and IM's got it bad. It's not dead yet, but it's got seven tentacles in the grave.

I came late to IM, so I've only now realized why the party is so quiet. I started with Beejive on my iPhone as an SMS alternative. It worked fairly well, though I ran into server disconnect and message delay problems. Then I started using it with Google Talk at work. There I ran into issues with messages going to one client or the other but not always both.

It wasn't until I started looking at multi-account desktop XP clients, however, that I realized how bad things were. That's where I found cr*pware bearing unwanted toolbars, neglected and buggy open source solutions, walled gardens from AOL and Microsoft, and web apps that want my google credentials (good luck with that).

Yee-uch. I know that smell!

So if IM is dead or dying, who held the knife?

I'm guessing it was a combination of Twitter, SMS/texting bundles, the mobile migration, the unflinching stupidity of Yahoo/AOL/Microsoft/Skype (basically everyone but Google), the non-multitasking iPhone and, above all, the complete absence of any plausible revenue stream [1].

IM, I barely knew ya.

[1] So why are there pretty-good IM clients on the iPhone? Hint.

--
My Google Reader Shared items (feed)

Window resizing - OS X vs. XP

On XP I can get "stuck windows" when I move my laptop between displays. These are windows that I can't resize, because they're too large for me to reach the right lower corner. (I think there are other causes of stuck windows.)

On OS X if I click the green "right size" button windows resize to fit the screen -- without going full screen. So they don't get stuck.

It's a small feature, but the sum of these small things is part of what makes Apple products a pleasure to use.

Alas, as is common these days, there are signs of regression to the lowest common denominator. iTunes doesn't work properly, and when Apple tried to make the "right size" button work correctly users rebelled and Apple reverted to the bad behavior (it creates a mini-player instead, you have to option-click to get it to work). Many apps uses to try to guess how to best use the display surface, but now they fill the screen -- which is absurd on a 27" monitor.

Does Windows 7 do anything clever here, or is stuck in the XP world?

--
My Google Reader Shared items (feed)

Haiti: Why I donate via CARE.ORG

During disasters like this it's common to donate to the American Red Cross. Obviously, a reputable place -- but you run the risk of getting spammed, mailed, phoned, etc.

CARE.ORG, a four (max rating) star charity, doesn't harass me - and I've been using them for at least six years. If they do email or contact you, tell them to put you on their no-contact-ever list. It works

They're active in Haiti ...

CARE: Donate Now:

... CARE is deploying additional emergency team members to the devastated city of Port-au-Prince in Haiti, where the worst earthquake in 200 years destroyed houses and left thousands homeless...

CARE will use your money well. Recommended.
--
My Google Reader Shared items (feed)

Innovations in comment spam

Comment spam continues its rapid evolution. Despite my reluctant surrender to the Captcha I'm seeing novel mutations every few months.

A recent technique is to write a reasonably detailed comment about a fairly specific topic, like "junk DNA". A query engine then identifies all blog posts that have a high match to the comment. An automated posting process, perhaps with some tool-assisted human powered captcha processors (via Amazon's Mechanical Turk?), submits the post to thousands of blogs.

Even with human review, the comment submissions will be a good quality match to a meaningful number of blog posts. The comment gets posted, and the spammers get something of value (link referrals?).

The one I rejected today was clumsily written, so it was fairly easy to spot. It contained an unnecessarily specific reference to a "first post", the author name was a marketing phrase, and the grammar and phrasing could have been better. I've probably missed better ones!

We can expect rapid improvement. In time they might evolve to transiently novel insights statistically applied to the right spot at the right time. At that point, would we not welcome them?

In the meantime we do need Google to start filtering these comments the same way they filter email. This particular approach lends itself to statistical filters, and of course the use of author reputation in filtering algorithms. Alas, Google has forgotten all about poor Blogger ...

--
My Google Reader Shared items (feed)

Brave new world: China attacks Google

Based on the phrasing and response, it's clear that Google believes this attack was launched by parties working for the government of China. We can also assume that the "relevant US authorities" (FBI) agree with them. I wonder if the targeted companies used software with similar vulnerabilities.

Official Google Blog: A new approach to China

Like many other well-known organizations, we face cyber attacks of varying degrees on a regular basis. In mid-December, we detected a highly sophisticated and targeted attack on our corporate infrastructure originating from China that resulted in the theft of intellectual property from Google. However, it soon became clear that what at first appeared to be solely a security incident--albeit a significant one--was something quite different.

First, this attack was not just on Google. As part of our investigation we have discovered that at least twenty other large companies from a wide range of businesses--including the Internet, finance, technology, media and chemical sectors--have been similarly targeted. We are currently in the process of notifying those companies, and we are also working with the relevant U.S. authorities.

Second, we have evidence to suggest that a primary goal of the attackers was accessing the Gmail accounts of Chinese human rights activists. Based on our investigation to date we believe their attack did not achieve that objective. Only two Gmail accounts appear to have been accessed, and that activity was limited to account information (such as the date the account was created) and subject line, rather than the content of emails themselves...

... We launched Google.cn in January 2006 in the belief that the benefits of increased access to information for people in China and a more open Internet outweighed our discomfort in agreeing to censor some results. At the time we made clear that "we will carefully monitor conditions in China, including new laws and other restrictions on our services. If we determine that we are unable to achieve the objectives outlined we will not hesitate to reconsider our approach to China."

These attacks and the surveillance they have uncovered--combined with the attempts over the past year to further limit free speech on the web--have led us to conclude that we should review the feasibility of our business operations in China. We have decided we are no longer willing to continue censoring our results on Google.cn, and so over the next few weeks we will be discussing with the Chinese government the basis on which we could operate an unfiltered search engine within the law, if at all. We recognize that this may well mean having to shut down Google.cn, and potentially our offices in China.

The decision to review our business operations in China has been incredibly hard, and we know that it will have potentially far-reaching consequences. We want to make clear that this move was driven by our executives in the United States, without the knowledge or involvement of our employees in China who have worked incredibly hard to make Google.cn the success it is today. We are committed to working responsibly to resolve the very difficult issues raised.

This may be the end of Google's services in China. We should expect their share price to fall in the morning. Google's "evil score" has now dropped to the lowest possible level for a public corporation.

Update 1/13/10: There's a lot of commentary this morning, including comparisons to how the USSR hobbled itself by shutting out access to world knowledge. I'm wondering if Google's increasingly powerful and ubiquitous machine translation services played a precipitating role. Language has been the cultural equivalent of the Himalayas - preserving China from cultural invasion. I suspect the Chinese government is very concerned about widespread direct unmediated access to English language materials.
--
My Google Reader Shared items (feed)

Dark matter DNA

Our universe is largely built with matter that shapes large structures, but doesn't interact with electric fields - including light. It's dark matter.

There's a funny similarity to our DNA ...

Borna Virus Discovered in Human Genome - Carl Zimmer - NYTimes.com

...Fossil viruses are also illuminating human evolution. Scientists estimate that 8.3 percent of the human genome can be traced back to retrovirus infections. To put that in perspective, that’s seven times more DNA than is found in all the 20,000 protein-coding genes in the human genome.

In the physican universe dark matter is only about 70% of all matter, but in humans "dark DNA" is 97%+ of all DNA. So our DNA is about 2% protein coding, 8% retrovirus, and 90% other - including non-retroviral virus origin and "structural". (Yes, I know that's "four times" and Zimer says "seven times" - his numbers are more likely correct.)

So from a DNA perspective, are we basically an ambulatory viral ecosystem with a fraction of information capacity that does things like make brains and bodies? Seems a bit much, but it turns out even some of the most important protein coding DNA is of viral origin. In a companion post on his blog Zimmer writes ...

... a virus protein called syncitin ... is essential for placentas to develop. Cells push the protein to their surface, where it lets them latch onto other cells, fusing together to create a special layer through which nutrients can pass from mother to child. The protein got its start on viruses, which use it to latch onto host cells and fuse to them, allowing their genes to slip in.

But recent research has revealed an intriguing new twist to our viral legacy. It turns out that the viral surface protein in question has a second job. It also tamps down the immune system of its host...

So is there any non-structural DNA in humans that's not of fundamentally viral origin?

See also: Presser on the bornavirus article ... UTA News Center

PS. A search on Preeclampsia and bornavirus has 180 hits today, but I think they appear to be loose and coincidental relationships. I didn't see research relating bornavirus-like superinfection triggering auto-immune placental disruption and thus pre-eclampsia / toxemia.

Update 1/30/2010: io9 quotes Frank Kelly: "[T]he human genome has evolved as a holobiontic union of vertebrate and virus... ". A Coral holobiont is "the entire community of living organisms that make up a healthy coral head".

--
My Google Reader Shared items (feed)

Lessons from my leonine chat icon

If you inspect my profile on various OS X and Google systems lately, you'll see a theatrical yawn ...

There's a lesson in the yawn. When I created a new user account on my i5 running 10.6, I chose a standard animal icon. Since it's a family machine, I wanted to choose an icon that would impress the children (didn't work). Hence the lion.

I then connected that account to my MobileMe account and, just as I found on 10.5 11 months ago, the login image on the iMac propagated to all my MobileMe associated machines, wiping out whatever I had there.

It ate them.

Then, after I fiddled with iChat and Adium, it propagated to Gmail and GoogleTalk/Video Chat and the wider world.

None of this is documented of course. It just happens. It's an emergent behavior; a side-effect. One bit of whimsy, and bam -- I'm a lion everywhere.

There will be more of these things in years to come. More strange leakages and propagations.

If you want something private, keep it on paper. And keep the paper out of range of Vicon Revue wearing lifebloggers ...

Update 1/12/10: Today I notice the OS X 10.6 lion has metastasized to my Google Reader Shared By ...

I'm sure this is violating all kinds of copyright laws, but all of my actions were entirely correct. I think I'll just have to get used to my emergent avatar. Maybe he'll appear on my virtual tombstone.

Update 1/18/10: Here it is on my Google Profile.

This is really silly. I'm going to try restoring the GP image and see if it propagates the other way.

Update 2/9/10: Now it's spread to Google Buzz.

Only it's no longer affixed by my gmail address, it's attached to my corporate email!

--
My Google Reader Shared items (feed)

If you're wondering where your money went ...

Still way down from the peak, almost 10 years later ...

Bubbleheads II - Grasping Reality with Opposable Thumbs:

...

S&P 500, June 30, 2000 close: 1455
S&P 500, December 31, 2009 close: 1145
Consumer Price Index, November 2009/June 2000: 1.26

Real price decline: -37.5%...

--
My Google Reader Shared items (feed)

How removing my car stereo gave me my Apple iSlate prediction

[Update: iPad is the name. My post-release verdict is even more flamboyant.]

Geeks are all tingly in the run up to Steve Jobs' iSlate/iPad/whatever announcement. The last time I remember this level of geek thrill was just before the Segway was announced.

Oh, you don't remember that? Well, it wasn't the Segue of a thousand jokes back then. It was a mysterious product that was going to transform the world. (Who knows, when gas is $12/gallon maybe it will.)

The Segway is a cautionary tale, but I'm rooting for Mr Jobs. Even his mistakes are interesting, and if anyone can make a slate exciting it's the man in the black shirt. Personally I'm much more interested in the $150 Chrome OS gBook, but I'll be tracking the fan sites nonetheless. I expect the slate to solve at least one problem I have, and to solve it in a way that will work for my iPhone and desktop too.

I expect Mr. Jobs to come up with a Digital Rights Management scheme for books that we can live with -- just as he (and his team) have done for video and apps. (BTW, do you think anyone notices that balanced DRM is the key to Apple's App Store windfall? The industry hasn't missed this, even though the media has.)

I want Apple to do this, because this morning I couldn't figure out how to get my ultra-geeky SONY car stereo out of my dying 1997 Subaru Legacy (we bought the Forester, not the whacked new Outback). I knew Crutchfield would have great directions, but they charge $10 for detailed directions unless you're buying a stereo -- and they US Mail them.

The price was a bit steep, but the real problem for me was US Mail. They do this, of course, because if they let users download a PDF they'd sell one copy of the directions.

What Crutchfield and I needed was a DRM approach that was a reasonable balance between their interests and mine. If they had that, they might sell the directions electronically for a more appealing $5.

That's my iSlate prediction. That Jobs/Apple will include a DRM solution for printed material that will, like their DRM for Apps, be a reasonable balance between the rights of publishers and the interests of consumers.

--

My Google Reader Shared items (feed)

Inbox zero - mastering email

I'm doing a 1 hour session on mastering email at my day job. I get to do this because, after 20 years of struggling with email, I have finally figured out how to do it.

For what it's worth I'll add a link to my presentation here after Jan 24th, but there's no great mystery to it. The most important intervention was reducing inflow. Of course I got rid of all email lists, newsletters and the like -- if an organization can't figure out blogs they're unlikely to have anything useful to tell me. Most of all though, I reduced the number of email replies and misdirected emails that I get.

I reduced the number of email replies by, paradoxically, spending more time crafting precise responses, and by being quicker to convert dysfunctional email to a meeting or phone call. I craft my response to an email so that no further correspondence should be necessary. If an email discussion goes beyond two cycles that's a meeting. It's almost always, in this context, a brief, productive, and satisfying meeting. The body of the meeting appointment, by the way, includes the last email sent. (In Outlook drag and drop the email on the calendar icon.)

I reduced the number of emails I had to reply to by gently educating my correspondents about what goes on the To line. The To line should include only people with tasks - such as the single person who should respond.

I reduced the time required to process and triage email by gently teaching about the correct use of the subject line. It should tell the reader what the email is about and what's needed. I change the subject line when I reply to precisely describe my replay -- including an answer summary. This subject line also makes my full-text search email archives more valuable.

These days the email I get is satisfying. It's increasingly well written, targeted, and easy to respond to. I'm now in a virtuous feedback loop; good email begets good email. (though example alone is not enough, cautious education is needed to).

More after the 24th of January.

See also some other posts of mine:

• Gmail's biggest missing feature - and it's a whopper.
• Getting Things Done and Managing Email with Lookout for Outlook (2004 - now I use Windows Search and my approach is a bit different in other ways)
• Beating email - it's doable. Here's how. (July 2008 - I've moved another notch up since that time)
• Inbox zero - zero - at last: In Nov 2010 I get to zero on my personal email. A slightly different path.

Update 11/8/10: Here's the presentation I promised. It should have all the corporate references expunged.

Bermuda

I came across Bermuda cruising the ocean floor on Google Earth...

There's a lot down there.

IOT: Samarkand, the Sogdians and the Silk Road

Once it was Maracanda, ruled by Alexandere. Centuries later, before Rome fell, the Persian speaking Sogdians flourished there, at the heart of the a historically trading empire that lasted from before 300 CE until after 700 CE. They were the traders of the Silk Road, and the conduits for Buddhism and much knowledge of China, India, Asia and places West.

Later their city became a place of Arab history - Samarkand.

Today Samarkand is in Uzbekistan ...

It's a hike, but it's a city of about 400,000 and it's open for tourism. In Google earth you can see their photos.

You can learn the story of the Sogdians, and a surprising amount of China's endless story by listening to ...

BBC - Radio 4 - In Our Time - The Silk Road

In 1900, a Taoist monk came upon a cave near the Chinese town of Dunhuang. Inside, he found thousands of ancient manuscripts. They revealed a vast amount of evidence about the so-called ‘Silk Road’: the great trade routes which had stretched from Central Asia, through desert oases, to China, throughout the first millennium....

Most of what we know of this people comes from a small cache of lost Sogdian mail, and the stories the Chinese told of the them. If not for that accident, we'd know almost nothing.

And yet, they changed the course of history.