Building a Web Scraper in Go: Understanding the Standard Library and Where x/net/html Fits

You can build a small, polite web scraper in Go with mostly standard-library code, but not entirely. The standard library covers the HTTP client, URL parsing, cancellation, and streaming reads. HTML parsing is the exception: the HTML5 tokenizer and tree parser live in golang.org/x/net/html, an external module you add with go get. The accurate description is therefore “standard library for fetching and URL handling, external module for parsing.”

Which package does what

Each part of a basic scraper maps to one package. Knowing the boundary matters because the external module is versioned separately from Go itself, so it appears in your go.mod file and needs its own upgrade decisions.

#	Preview	Product	Price
1		Web Scraping with Python: Collecting More Data from the Modern Web	$27.89	Buy on Amazon
2		Web Scraping with Python: Data Extraction from the Modern Web	$44.94	Buy on Amazon
3		Hands-On Web Scraping with Python: Extract quality data from the web using effective Python...	$46.99	Buy on Amazon
4		Web Scraping with Python: Collecting Data from the Modern Web	$19.09	Buy on Amazon
5		Python Web Scraping Cookbook: Over 90 proven recipes to get you scraping with Python, micro...	$21.68	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Fetching a page safely

Fetching is where most scrapers go wrong: they run with no timeout, leak connections, or accept whatever arrives. The steps below use only standard-library packages.

Create one reusable client with a timeout

The default helper http.Get uses http.DefaultClient, which sets no timeout. For a scraper, create your own client once and pass it around. The Go Authors’ net/http package documentation (see the net/http source documentation) states: “Clients and Transports are safe for concurrent use by multiple goroutines and for efficiency should only be created once and re-used.”

#1 Best Overall

Sale

Web Scraping with Python: Collecting More Data from the Modern Web

$27.89

Buy on Amazon

Concurrency safety of the client is not permission to send unlimited requests to a site. It only means that goroutines can share the client, which is covered in the politeness section below.

A Client.Timeout value covers the whole exchange, including connection setup, redirects, and reading the response body. Set it to a value that suits the target site; 15 seconds is a reasonable starting point for ordinary HTML pages, not a documented default.

Attach a context to every request

Create requests with http.NewRequestWithContext rather than http.NewRequest. Pass in a context from context.WithTimeout or context.WithCancel, so that the crawl as a whole can stop and any in-flight request is cancelled when the context ends. A single context for the whole run is the simplest budget; per-request contexts are useful when some pages should get shorter deadlines.

Set headers explicitly

Set a User-Agent that identifies your tool and gives the site operator a way to contact you. Use req.Header.Set for this and for any Accept header. Headers belong on the request object, not on the client, when they vary per request.

Rank #2

Sale

Web Scraping with Python: Data Extraction from the Modern Web

$44.94

Buy on Amazon

Check the status, the content type, and close the body

The net/http documentation requires you to close the response body when you are finished with it. Put defer resp.Body.Close() directly after the error check on client.Do, so the body is closed on every path, including error statuses.

• Status: accept only the codes your logic expects. The example below accepts 200 OK and records other statuses as skipped pages.
• Content type: check the Content-Type header before parsing. Skip anything that is not text/html, such as PDFs or images linked from a page.
• Redirects: the client follows redirects by default. Use resp.Request.URL to learn the final URL after redirects so that relative links resolve against the right base.

Bound what you read

Do not assume every remote response is small. Wrap the body in io.LimitReader with one byte more than your limit, then reject the page if it returned more than the limit. The 2 MiB cap in the example is an illustrative choice, not a standard. For error responses you do not need to parse, read only a small bounded amount before closing.

Parsing HTML: tree or token stream

golang.org/x/net/html offers two approaches. Choose by what your extraction needs, not by which one looks more complete.

html.Parse for a document tree

html.Parse builds a full node tree using HTML5 tree-construction rules. It is convenient when you want to walk the document, find an element, then look at its children and ancestors. The trade-off is memory: the whole document is held as a tree before you can query it.

Rank #3

Hands-On Web Scraping with Python: Extract quality data from the web using effective Python techniques

$46.99

Buy on Amazon

Two package-level constraints are worth knowing. The package documentation says that malformed input may produce implied, moved, or dropped nodes, so the tree is the parser’s reconstruction rather than a copy of your markup. It also says the input is assumed to be UTF-8 and that nesting deeper than 512 elements is rejected. If a site serves another encoding, decode it to UTF-8 before parsing; the parser does not do that for you.

html.Tokenizer for streaming extraction

html.NewTokenizer reads one token at a time: start tags, end tags, text, comments, and errors. It is a good fit for pulling a title or a few attributes out of a large page without building a tree. Be careful with byte slices. The slices returned by z.Text() and z.Raw() may be overwritten by the next call to z.Next(), so convert them to string immediately if you keep them.

Trust what the parser produces

The package warns that parsing and interpreting untrusted HTML needs care. Do not treat the parsed structure as an exact reproduction of the source nesting, and do not make security or trust decisions from the tree alone. For a scraper that only collects links and text, this is a correctness point, not a reason to avoid the parser.

Resolving and filtering links

Extracted links are often relative (/articles/42, ../about). Resolve each one against the URL of the page where you found it, not the start URL. url.URL.ResolveReference does this correctly, and it is the reason to prefer net/url over string concatenation.

Rank #4

Sale

Web Scraping with Python: Collecting Data from the Modern Web

$19.09

Buy on Amazon

After resolution, apply your own policy before adding a link to the queue. The package documentation gives you the URL utilities; the policy is yours. A reasonable minimum is:

• keep only http and https schemes;
• keep only hosts you intend to crawl, comparing the full host including any port;
• remove fragments so that /page#a and /page#b are not fetched twice;
• skip URLs already seen, using a map keyed by the normalized string.

Choosing between the options

Use this table to decide which approach fits a given job. Neither option is universally better.

Complete example: a bounded, same-host crawler

The program below fetches a start URL, follows same-host links breadth-first, prints each page’s final URL and title, and stops after 20 pages. It is sequential and waits one second between requests. It uses only net/http, net/url, context, and io from the standard library, plus golang.org/x/net/html.

Set up the module

1. Create a directory and enter it: mkdir scraper && cd scraper
2. Initialize the module: go mod init scraper
3. Add the HTML module: go get golang.org/x/net/html. Check go.mod afterwards and keep the version your project pins.
4. Save the code below as main.go.
5. Run it against a site you are permitted to crawl: go run . https://example.com/

Each output line is a tab-separated final URL and page title. The program prints nothing for pages it skips, and it reports skipped pages on standard error.

Best Value

Sale

Python Web Scraping Cookbook: Over 90 proven recipes to get you scraping with Python, micro services, Docker and AWS

$21.68

Buy on Amazon

main.go

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"os"
	"strings"
	"time"

	"golang.org/x/net/html"
)

const (
	maxBodyBytes = 2 << 20 // 2 MiB cap per page (illustrative)
	maxPages     = 20
	crawlDelay   = 1 * time.Second
	userAgent    = "example-scraper/1.0"
)

// fetchPage GETs a URL and returns the final URL after redirects and the body.
func fetchPage(ctx context.Context, client *http.Client, raw string) (*url.URL, []byte, error) {
	target, err := url.Parse(raw)
	if err != nil {
		return nil, nil, err
	}
	if target.Scheme != "http" && target.Scheme != "https" {
		return nil, nil, fmt.Errorf("unsupported scheme in %q", raw)
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, target.String(), nil)
	if err != nil {
		return nil, nil, err
	}
	req.Header.Set("User-Agent", userAgent)
	req.Header.Set("Accept", "text/html")

	resp, err := client.Do(req)
	if err != nil {
		return nil, nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))
		return nil, nil, fmt.Errorf("GET %s: %s", target, resp.Status)
	}
	if ct := resp.Header.Get("Content-Type"); !strings.HasPrefix(ct, "text/html") {
		return nil, nil, fmt.Errorf("GET %s: unexpected Content-Type %q", target, ct)
	}

	body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes+1))
	if err != nil {
		return nil, nil, err
	}
	if len(body) > maxBodyBytes {
		return nil, nil, fmt.Errorf("GET %s: body larger than %d bytes", target, maxBodyBytes)
	}
	return resp.Request.URL, body, nil
}

// extractLinks returns absolute http(s) links from anchor tags, fragments removed.
func extractLinks(page *url.URL, body []byte) ([]string, error) {
	doc, err := html.Parse(bytes.NewReader(body))
	if err != nil {
		return nil, err
	}
	var links []string
	var walk func(*html.Node)
	walk = func(n *html.Node) {
		if n.Type == html.ElementNode && n.Data == "a" {
			for _, a := range n.Attr {
				if a.Key != "href" {
					continue
				}
				ref, err := url.Parse(strings.TrimSpace(a.Val))
				if err != nil {
					break
				}
				abs := page.ResolveReference(ref)
				if abs.Scheme == "http" || abs.Scheme == "https" {
					abs.Fragment = ""
					links = append(links, abs.String())
				}
				break
			}
		}
		for c := n.FirstChild; c != nil; c = c.NextSibling {
			walk(c)
		}
	}
	walk(doc)
	return links, nil
}

// firstTitle streams tokens until it finds the