User-agent: * Crawl-delay: 60 Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /medias/ # Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. User-agent: Twitterbot Allow: /images User-agent: facebookexternalhit Allow: /images