Webmaster Help Feed

http://tencell.com/15425-p-1.html is a duplicate of http://www.tencell.com/15425-p-1.html

canonicalization problem

And no, it's most certainly NOT an canonicalization problem.

Who the heck who said this question is answered???  Is there any way to find that out? Is that what the Forums are about? Taking any answer as answered so valid questions Google can't--or won't--answer just die on the vine?  That's beyond childish.

Hi Tencell,

Check here(http://www.google.com/support/forum/p/Webmasters/thread?tid=1b84082a9efb1209&hl=en) and use the snippet of python code John Mu provides to find any duplicates.

Hope this helps.

There IS no duplication, glendene. It's a Google MISTAKE that they simply keep ignoring, hoping we'll all go away and leave them to keep screwing with our indexed page results in various parts of the country and the globe to favor people that pay them for search results. Google's been making thousands of mistakes this year--huge and small.  The common denominator is that this year they're not admitting to ANY of them.  Apparently from this point forward in Google's corporate history, all Google mistakes, blunders, screwups, and crappy programming are the fault of the webmasters employing Google for search traffic or monitoriing their sites through Google's newly abominable Webmaster Tools. 

But as you've probably discovered, along with the rest of us: Google's Webmaster Tools sitemap configuration module is the only place in their newly revised Webmaster Tools that shows the actual number of your site's pages that they've included in their indexes. There is no other accurate means of assessing your progress in having your pages returned to Google's index after yet another one of their index-dumping miscalculations, is by monitoring that sitemap module in Webmaster Tools.

Dnyhagen, you can always fall back on site:www.sitename.com to see roughly what's been indexed

Uh, no, 1918, as you WELL KNOW. The ONLY accurate measure FROM GOOGLE is GOOGLE's measure of the thousands of perfectly valid--and historically valuable--pages they've dumped en masse from their index from time to time recently only to trickle them back into their indexes at the rate a 5 to 6 reindexed pages a month.  All this rubbish about the sitemap subsection of Webmaster Tools being useful--or meaningless for that matter--is pure rubbish.  You know it and Google knows it.

Should you need any proof, simply note the utter and complete absence of any response from Google over this latest Webmaster Tools fiasco--in a long recent string of Webmaster Tools fiascos.

Then of course we could all draw the conclusion that if, as Google reps repeat here time and time again, the sitemap subsection of Webmaster Tools is meaningless, that Google's programmers were either:

1. Idiots for incorporating an utterly meaningless module into Webmaster Tools in the first place.
2. At the least, disengenuous, for telling us that the sitemap module serves any valid purpose whatsoever.
3. Covering Google's corporate patoot by going through the motions of appearing to correct a host of their incompetent missteps from the recent past.
4. Duped just like the rest of us into believing that Google is honestly attempting to make their search results more valid and valuable to searchers without either regional throttling to skew the results, favoring millions of AdSense and AdWord subscribers, or giving the impression of favoritism to either Web 2.0 clients, social networking sites, or blog sites over traditional websites.

Google's continued silence of this issue--and a host of other recent issues--simply raises more questions about Google's current competence, integrity and validity than it answers.

Sorry to be the one of the few to notice that The Emperor is butt nekid of late, but that's just me talking . . .

Heh, Bingo! davebarley! You're almost certainly correct, Sir!

All that remains now is for Google to send out a form letter to every sitemap.xml generating program supplier, that Google's made a teensy change to their Webmaster Tools that will now entail hundreds of them to re-write their sitemap.xml preparation programs to account for this latest 'beneficial tweak' that Google's made to their Webmaster Tools Sitemap module.

Simple, no?

From looking at your sitemap.xml you're directing it to the numerous text files you're employing for sitemaps. I'm sure you've already gone through this exercise, but have you made absolutely certain that none of the extremely long aggregations of part numbers you're listing as pages aren't duplicated somewhere in one of those text files? 

I noted that you've at some point or another employed, for example, a ''1-'' prefix for many of your parts that seem in line with the none '1-'' prefixed similar part numbers? Perhaps you could throw together a little excel function to double check?

I can sympathize with your problem.  Your sitemap text files look to be a daunting preparation issue.

Are they all really pages you need to point Google to?  (e.g., as opposed to simply letting Google find them on their own.)

I don't mean to oversimplify the your very valid issue.  Just talking out loud here . . .

schmidtpainting, you're submitting both http://www.schmidtpainting.net/index.html  and http://www.schmidtpainting.net/ in your Sitemap file. This doesn't make sense since it's really the same page. If you submit only one of them, the warning will go away. Please note that this message in Webmaster Tools does not signal a change in the way we've been processing your Sitemap file.

Cheers
John

Hi TenCell, can you confirm that you haven't changed your Sitemap text files since seeing this message? I think that's the case, but just want to confirm it to be certain. I've passed your Sitemap files on to the team and will get back to you once I know more. Thanks for your patience.

Cheers
John

Hi Funksen, you've got the URL "http://tech.gate.io/blogpost12" listed twice in your Sitemap file. If you remove one of those lines, the message will disappear. You should probably be seeing this URL in the message in your Webmaster Tools account as well.

Cheers
John

Anyone else lose all their indexed URLs the day after making this duplicate index.html vs. canonical fix by removing one or the other? I had 1200 indexed URLs before the duplicate removal.  Now only 2.

None of that seems to have worked for me:

I changed the 301-redirect on the server to point index.html to www.digitaldeliftp.com

I ensured that my sitemap location in robots.txt was the www.digitaldeliftp.com location.

I regenerated my sitemap.xml with all www.digitaldeliftp.com prefixes

I removed the 'duplicate' www.digitaldeliftp.com/index.html that coffeecup software has been incorporating into my sitemap.xml for over two years now without consequence.

I resubmitted my sitemap.xml, got the green check mark and I'm still back to 2 URLs indexed from the previous 1200 that took me the last six months to get back to from the over 2600 I'd had indexed historically going back as long as two years.

Must still be doing something wrong, obviously.  But what?

@Dnyhagen: While this message is newly visible in Webmaster Tools, the whole processing of Sitemaps files has not changed recently. If something else happened at the same time as the message showed up in your account, they generally aren't related.  I'll check out your other threads in a bit to see if there's something I can add there.

@Johnny Ramone: If both http://www.example.com/ and http://www.example.com/index.html show the same content, there's no reason to submit both via Sitemaps (since we'll pick one and hide the other one in search results anyway). Personally, I'd pick the shorter URL, but the choice is ultimately up to you.

Cheers
John

@Johnny Ramone: Sure, leave it like that if you prefer. However, the advice to pick a canonical version is pretty good and has been around for a LONG time :-). For example, Matt Cutts mentioned it back in 2006 in http://www.mattcutts.com/blog/seo-advice-url-canonicalization/

Personally, I'd choose a canonical URL and remove the duplicate, but I can understand that you might want to be careful and wait to see what all happens. Search engines are strange "black boxes" sometimes and taking a step back to wait for things to settle down is almost never bad advice.

When you're ready, if you want, you can follow up your choice of URLs by using the rel=canonical link element. This gives us another signal to let us know which version you prefer to have indexed. You can find out more about it in our blog post at  http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

Cheers
John

@ JohnMu thanx

Hi budster
You're right - this message is new. However, the processing in the background has not changed. We've always treated these URLs as duplicates, but we're now exposing this to our users so that they can learn from this as well. Removing duplicates from a Sitemap file always makes sense provided you leave one copy. It will not affect your page's rankings in a negative way because we've always filtered out the duplicate version anyway.

Cheers
John

pssstttt. .  . budster. . . . you might want keep reality checks on the down low here in the forums. The skins here are as delicate as precious young orchid buds for one, and any hint or mere mention of how things work in the real business and professional world outside of Google is frowned upon.  To prepare yourself to post your next comment, just remember to genuflect in the direction of Mountain View, keep real world common sense to yourself, and never ever ever suggest that Google may have overreached a teensy bit from time to time. The last thing you want to do here is get shunned for being honest and candid. Think GroupThink and you'll get along just dandy.

Hi TenCell
It's good to hear that things are looking up with regards to your Sitemap files :-). It's possible that the engineers will remove this message for the moment anyway, since it seems to have confused a lot of people (the processing in the background will remain the same though).

Regarding the indexing of your URLs, that's a bit more complicated, especially when the site has a size like yours. One element which is always important though is that URLs should be well-connected with links, either from other pages within your site or from other websites.

With that in mind, I took a random URL from your seventh Sitemap file and took a quick look at it: http://www.tencell. com/88970422-p-300125.html -- apart from the fact that it looks like there is very little unique and compelling content on the page (I know this is a problem with some article databases, perhaps the part number alone is the most important part, you'd know best), it appears that this URL is not linked anywhere from your website, nor does it link to other articles on your site (eg. related components, groups or sub-groups). This makes it very hard for us the judge the relevance of a single URL like that and in the end, it makes it hard for us to judge how important it is that this particular URL should be indexed like that.

So my advice would be to not rely completely on Sitemap files but also to make sure that your content can be found with links within your website -- and that content on your website links out to similar content within your site. Also, adding a bit of unique and compelling content to the article pages is generally a good idea, if there are ways that you could do that :-).

Hope it helps!
John

Hi TenCell
I followed up with the team regarding your Sitemap files and it does look like there was an issue on our side there. Sorry about the confusion. It seems this was only temporary though so it would have cleaned itself up after the next processing (which we pushed forwards a bit to get this error out of your dashboard).

Regarding indexing of your Site, one thing you could do is to provide overview pages for your parts. These pages could link to the individual part pages and contain a bit more information about the parts that they link to. In an attempt to keep things simple, it might even make sense to only submit "content-rich" pages via Sitemaps, especially if you are manually generating your Sitemap files at the moment. It's always hard working out the best configuration for a site like yours, so I'd keep a good eye on the users and try to make it as easy as possible for them, they're frequently the best sources of inspiration (it works here in the forum for us as well :-)).

@everyone: We've currently disabled these messages since it did confuse quite a number of users. We are however still processing Sitemaps files the way we've been doing so in the past, which means that we will still generally regard "/" and "/index.htm" as duplicates internally (but we won't bother you with messages - at least for the moment).

I would still recommend only submitting either "/" or "/index.htm" in the Sitemap file if both URLs lead to the same content. It won't make or break your website, but clean canonicalization is always a good practice. For more information on this canonicalization stuff, I'll re-post the links from above + a few others which I think are pretty good:
- http://www.mattcutts.com/blog/seo-advice-url-canonicalization/
- http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
- http://googlewebmastercentral.blogspot.com/2007/09/google-duplicate-content-caused-by-url.html
- http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html

Cheers
John

Hi budster
Which Sitemap generator do you mean - the older Python one or the newer one for Apache/IIS? I imagine changing this behavior in the Python one is fairly easy (though to be honest, I haven't done anything with it for quite some time now). If this is for the newer Apache/IIS one, I'd recommend posting about the issue in detail (with [anonymised] examples, if you have any) in the Google Group for it. Otherwise, feel free to just open a bug if you don't see a matching one already.
Thanks!
John

Well put, budster. One can't help but wonder, what with the incredible resource that these webmaster forums could be, that simple reality checks--from any level contributor--are predominantly met with either disdain or outright derision. JohnMu, God bless him, seems the most grounded contributor in the entire forum. At the very least we're all finally on the same page. Thanks for your contributions, clarifications, and sanity checks--both of you.