sscoble/brand-site
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

No sitemap.xml or robots.txt — both return 404 #38

New issue
Closed
opened 2026-05-02 05:10:03 +00:00 by sscoble · 2 comments
sscoble commented 2026-05-02 05:10:03 +00:00
Owner
Copy link

Found: /sitemap.xml → 404. /robots.txt → 404.

Gap: Search engines cannot efficiently discover or crawl pages. No crawl instructions exist at all.

Needs:

  • Generate sitemap.xml from Astro content collections (astro-sitemap integration)
  • Create robots.txt (allow all, point to sitemap URL)
  • Submit sitemap to Google Search Console
  • Submit sitemap to Bing Webmaster Tools
  • Add canonical URL tags to every page
  • Add meta robots on staging/draft content
**Found:** /sitemap.xml → 404. /robots.txt → 404. **Gap:** Search engines cannot efficiently discover or crawl pages. No crawl instructions exist at all. **Needs:** - Generate sitemap.xml from Astro content collections (astro-sitemap integration) - Create robots.txt (allow all, point to sitemap URL) - Submit sitemap to Google Search Console - Submit sitemap to Bing Webmaster Tools - Add canonical URL tags to every page - Add meta robots on staging/draft content
sscoble added this to the 📋 V2 milestone 2026-05-02 05:10:03 +00:00
agent-engineering commented 2026-05-05 22:01:14 +00:00
Collaborator
Copy link

Verified complete - robots.txt and sitemap.xml exist. robots.txt at /robots.txt returns valid content. sitemap.xml generated dynamically at /sitemap.xml with 6 pages (home, speaking, consulting, books, contact, privacy). Both accessible and functional.

Verified complete - robots.txt and sitemap.xml exist. robots.txt at /robots.txt returns valid content. sitemap.xml generated dynamically at /sitemap.xml with 6 pages (home, speaking, consulting, books, contact, privacy). Both accessible and functional.
agent-engineering commented 2026-05-05 22:06:34 +00:00
Collaborator
Copy link

VERIFIED COMPLETE - robots.txt and sitemap.xml exist. robots.txt at /robots.txt returns valid content. sitemap.xml generated dynamically at /sitemap.xml with 6 pages. Both accessible.

VERIFIED COMPLETE - robots.txt and sitemap.xml exist. robots.txt at /robots.txt returns valid content. sitemap.xml generated dynamically at /sitemap.xml with 6 pages. Both accessible.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
content/dark-factories-backfill
feat/seo-schema-improvements
content/agentic-wars-blog
content/blog-voice-rewrite-all-5
fix/blog-visual-structure
fix/brand-links-2026-05-06
fix/seo-meta-38-39-40
fix/67-hero-cta
pr-4-content-trust
fix/technical-51-55
fix/mobile-a11y-53-65
fix/talk-pages-58
fix/63-64-book-covers
pr-3-footer-cta
pr2-books-waitlist
pr1-hero-cta-fixes
close-resolved-issues
No results found.
Labels
Clear labels   
a11y

Accessibility

  
blocked

   
bug

Something broken or incorrect

  
content

Content writing, migration, editing

  
content

Content additions or updates

  
conversion

CTAs, forms, signup flows

  
design

Visual design, layout, branding

  
feature

   
improvement

   
infrastructure

Site infrastructure, tooling, hosting

  
infrastructure

Build, deploy, CI/CD, hosting

  
landing-page

Landing page deliverable

  
priority:critical

Site is broken or misleading without this

  
priority:high

Important for launch credibility

  
priority:low

Nice to have, polish item

  
priority:medium

Should have before full launch

  
question

   
seo

Search engine optimization

No labels
a11y
blocked
bug
content
content
conversion
design
feature
improvement
infrastructure
infrastructure
landing-page
priority:critical
priority:high
priority:low
priority:medium
question
seo
Milestone
Clear milestone
No items
No milestone
📋 V2
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sscoble/brand-site#38
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?