惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

aimingoo的专栏
aimingoo的专栏
B
Blog RSS Feed
Recent Announcements
Recent Announcements
Vercel News
Vercel News
M
MIT News - Artificial intelligence
阮一峰的网络日志
阮一峰的网络日志
L
LangChain Blog
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Microsoft Security Blog
Microsoft Security Blog
H
Help Net Security
T
The Blog of Author Tim Ferriss
Y
Y Combinator Blog
G
Google Developers Blog
罗磊的独立博客
爱范儿
爱范儿
宝玉的分享
宝玉的分享
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
酷 壳 – CoolShell
酷 壳 – CoolShell
博客园_首页
S
SegmentFault 最新的问题
WordPress大学
WordPress大学
月光博客
月光博客
人人都是产品经理
人人都是产品经理
Apple Machine Learning Research
Apple Machine Learning Research

Red Blob Games: latest blog posts

Red Blob Games: Differential heuristics Red Blob Games: Responsive design calculator Red Blob Games: Academic citations Red Blob Games: Highlighting interactive code blocks Red Blob Games: Optimizing page size Red Blob Games: Writing a guide to SDF fonts Red Blob Games: URLs with trailing punctuation Red Blob Games: What I did in 2025 Red Blob Games: Goodbye Sass Goodbye Sass Red Blob Games: RotMG map seeds Red Blob Games: Mapgen4 Mapgen4's use of WebGL2 Red Blob Games: Mapgen4 river shader Red Blob Games: Mapgen4 renderer Red Blob Games: Harnessing ChatGPT hallucinations Red Blob Games: Let Let's write a search engine, part 2 of 2 Let's write a search engine, part 1 of 2 Red Blob Games: Hexagon conversions Red Blob Games: Mapgen4 trade routes Red Blob Games: De-optimizing mapgen4 Red Blob Games: Hexagon spiral coordinates Red Blob Games: Thoughts on Flash Red Blob Games: What I did in 2024 Red Blob Games: Hexagon page animations Red Blob Games: SDF Halos SDF Halos Red Blob Games: SDF headless tests, part 2 Red Blob Games: SDF headless tests, part 1
Red Blob Games: Let
2025-08-30 · via Red Blob Games: latest blog posts

I have a search box on my web site that uses Google to search my site only using the site: operator[1]. Try it — type in hexagon tiles and you’ll see that it shows my hexagon tile page.

I wondered though: what would it take to make my own search feature that worked as you typed? This is what I’ll implement:

Prototype of search feature

I also wondered if there are other things I might want to build with this data, like Simon Willison’s “related pages” feature[2]. Or browsing by category. Or browsing by date.

The first thing I need to do is extract the text of the pages I want to search, without formatting or images or html. The page might contain this xhtml tree:

<p>
  <em>
    Graph search
  </em>
  algorithms let us find the shortest path on a map 
  represented as a graph. One popular algorithm is
  <a href="…">
    A*
  </a>
  .
</p>
<p>
  Let's start with
  <a href="…">
    Breadth First Search
  </a>
  and work our way up from there.
</p>

For searching, I’d like to simplify this into a set of lines, each representing one “block” of text:

Graph search algorithms let us find the shortest path on a map represented as a graph. One popular algorithm is A*.
Let's start with Breadth First Search and work our way up from there.

I can then run a tool like grep on these lines. Will this produce good search results? I don’t know! I need to try it and see.

I started thinking about the extraction algorithm from xhtml to text. Which tags should I consider “block”? I started looking at MDN[3] for a list. I also wondered what to do about situations like this:

<div>
  This is a block of text
  <div>
    inside another
  </div>
  block of text
</div>

Should it be

This is a block of text inside another block of text

or

This is a block of text
inside another
block of text

But I realized that trying to figure this out is a distraction. Maybe it’ll matter. Maybe it won’t. But I don’t know yet, and I shouldn’t prematurely try to solve this problem. Instead, the most important thing is to investigate the biggest unknown. The biggest unknown in this case: is searching over lines even useful?

So I instead tried this (using the xhtml input file, which is the page without the header, nav bar, footer, copyright, etc.):

cat pathfinding/a-star/introduction.bxml \
    | pandoc --read html --write plain --wrap=none \
    | less

I looked at the output and decided this is a reasonable starting point! It gives me the text of the page. I can refine it later.

The next step is to collect the text from all my pages:

fd --extension bxml | while read -r file; do
    cat "$file" | pandoc --read html --write plain --wrap none
done >all-pages.txt

That’s only 1.6MB. That could easily be loaded into a web browser. There are many web pages bigger than that!

I made a few changes to my shell script:

  1. Added a separator with the filename
  2. Skipped lines that don’t have text (pandoc outputs some blank lines)
fd --extension bxml | while read -r file; do
    echo "%%%% $file %%%%";
    cat "$file" \
        | pandoc --read html --write plain --wrap none \
        | grep '[a-zA-Z0-9]'
done >all-pages.txt

Great! What’s next? Let me load this into a web page:

const words = await (await fetch("./all-pages.txt")).text();
const lines = words.split("\n");

Now I can search over these lines.

function search1(event) {
    let results = [];
    let query = event.currentTarget.value;
    if (query) {
        for (let line of lines) {
            if (line.indexOf(query) >= 0) results.push(line);
            if (results.length > 10) break;
        }
    }
    document.querySelector("#search-results-1").textContent = results.join("\n");
}
document.querySelector("#search-query-1").addEventListener('input', search1);

Type hexagon into this search box:


  
Search engine 1:

Ok, great, this works and is fast! I don’t have to worry about building an inverted index or other optimizations. But it’s hard to see the matches. Let’s add:

  1. highlight the matches
  2. trim extra whitespace from the document
  3. case insensitive match
function escapeHTML(text) {     const map = {
        '&': '&amp;',
        '<': '&lt;',
        '>': '&gt;',
        '"': '&quot;',
        "'": '&#39;'
    };
    return text.replace(/[&<>"']/g, m => map[m]);
}

function search2(event) {
    let results = [];
    let query = event.currentTarget.value;
    if (query) {
        query = new RegExp(RegExp.escape(query), 'gi');
        for (let line of lines.map(line => line.trim())) {
            let matches = line.matchAll(query).toArray();
            if (matches.length > 0) {
                let result = "";
                let pos = 0;
                for (let m of matches) {
                    result += escapeHTML(line.slice(pos, m.index)) + "<b>" + m[0] + "</b>";
                    pos = m.index + m[0].length;
                }
                result += escapeHTML(line.slice(pos));
                results.push(result);
            }
            if (results.length > 10) break;
        }
    }
    document.querySelector("#search-results-2").innerHTML = results.join("\n");
}
document.querySelector("#search-query-2").addEventListener('input', search2);

Type hexagon into this search box:


  
Search engine 2:

Hey, that’s looking pretty good. We’re matching lines.

But I want to match documents. I had inserted document separators %%%% into the text file. Let’s use that now.

function parseIntoDocuments(text) {
        const results = text.split(/%%%% (.*?) %%%%$/gm); 
    if (results[0] !== "") throw `expected first split to be empty ${results[0]}`;
    if (results[1].length > 100) "expected second split to be filename";
    let documents = [];
    for (let i = 1; i+1 < results.length; i += 2) {
        let filename = results[i];
        let lines = results[i+1].split("\n");
        lines = lines.map(line => line.trim());         lines = lines.filter(line => line);         documents.push({filename, lines});
    }
    return documents;
}
const documents = parseIntoDocuments(words);

function search3(event) {
    const MAX_DOCUMENTS = 5;
    const SNIPPET_LENGTH = 3;
    let results = [];     let query = event.currentTarget.value;
    if (query) {
        query = new RegExp(RegExp.escape(query), 'gi');
        for (let document of documents) {
            let snippet = [];             for (let line of document.lines) {
                let matches = line.matchAll(query).toArray();
                if (matches.length > 0) {
                    let result = " … ";
                    let pos = 0;
                    for (let m of matches) {
                        result += escapeHTML(line.slice(pos, m.index)) + `<b>${m[0]}</b>`;
                        pos = m.index + m[0].length;
                    }
                    result += escapeHTML(line.slice(pos));
                    snippet.push(result);
                }
                if (snippet.length > SNIPPET_LENGTH) break;
            }
            if (snippet.length > 0) results.push({document, snippet});
            if (results.length > MAX_DOCUMENTS) break;
        }
    }
    let html = results
         .map(({document, snippet}) => `<i>[${document.filename}]</i>\n<small>${snippet} …</small>`)
         .join("\n");
    document.querySelector("#search-results-3").innerHTML = html;
}
document.querySelector("#search-query-3").addEventListener('input', search3);

Type hexagon into this search box:


  
Search engine 3:

Great! We’re now grouping matches into documents, and limiting how many show up.

But we’re showing the first N matching documents. This is like Ctrl + F over all documents. A “search engine” should show the best documents instead of the first in some arbitrary order. Let’s attempt that in the next post.