'\" t .\" Automatically generated by Pandoc 3.7.0.2 .\" .TH "bombadil" "1" "" "" .SH Introduction Bombadil is property\-based testing[1] for user interfaces. It autonomously explores and validates correctness properties, \f[I]finding harder bugs earlier\f[R]. It runs in your local developer environment, in CI, and inside \c .UR https://antithesis.com/ Antithesis .UE \c \&. .SS Why Bombadil? Or rather, \f[I]why property\-based testing?\f[R] Because example\-based testing, especially when browser testing, is costly and limited: .IP \(bu 2 Costly, because maintaining suites of Playwright or Cypress tests takes a lot of work. Even in the age of AI, tests written and updated by coding agents can easily break and require your attention. .IP \(bu 2 Limited, in that they only test very small parts of the state space; a bunch of happy cases, a set of regression tests, and maybe even some error handling cases that are important. But what about everything else \(em like all the stuff you or the agent didn\(cqt think about testing? .PP This is where property\-based testing, or \f[I]fuzzing\f[R] if you will, comes into play. By randomly and systematically searching the state space, Bombadil behaves in ways you didn\(cqt think about testing for; unexpected sequences of actions, weird timings, strange inputs that you forgot could be entered. .SS How it works Instead of describing \(lqwhat good looks like\(rq in terms of fixed test cases, you express general properties of your system, defining how it should behave in all cases. Bombadil checks each property as it explores your system in its chaotic ways, reporting back any violations. .PP To test a web application using Bombadil, you write a specification in TypeScript that exports properties and action generators. These can be domain\-specific \(em to exercise and validate your system\(cqs logic in custom ways \(em or be imported from Bombadil\(cqs defaults. Bombadil tests anything that uses the DOM, no matter how it\(cqs built. This includes single\-page apps, server\-side rendered apps, and even static HTML. .PP Conceptually, it runs in a loop doing the following: .IP "1." 3 Extracts the current state from the browser .IP "2." 3 Checks all properties against the current state, recording violations[2] .IP "3." 3 Selects the next action based on the current state, and performs it .IP "4." 3 Waits for the next event (page navigation, DOM mutation, or timeout) .IP "5." 3 \f[I]Returns to step 1\f[R] .PP Bombadil itself decides what is an interesting event and when to capture state. You provide the properties and actions, Bombadil does the rest! .SH Getting started Bombadil runs on your development machine if you\(cqre on macOS or Linux. Use it to validate changes to TypeScript specifications, and to run short tests while working on your system. Configure something like GitHub Actions or a cron job to continously run longer tests on your main branch or in a deployed staging environment. When you\(cqre ready to put your system under extreme conditions, and test your full stack deterministically, run it inside \c .UR https://antithesis.com Antithesis .UE \c \&. .SS Installation The most straightforward way for you to get started is downloading the executable for your platform: npm .PP Install as a development dependency in your project: .IP .EX npm install \-\-save\-dev \(atantithesishq/bombadil .EE .PP Add a script to your \f[CR]package.json\f[R] to run Bombadil: .IP .EX { \(dqscripts\(dq: { \(dqtest\(dq: \(dqbombadil browser test \-\-time\-limit=1m https://your\-app.example.com\(dq } } .EE .PP Then run it with \f[CR]npm test\f[R]. This also provides TypeScript type definitions for writing specifications. macOS .PP Download the \f[CR]bombadil\f[R] binary using \f[CR]curl\f[R] (or \f[CR]wget\f[R]) and make it executable: .IP .EX curl \-L \-o bombadil https://github.com/antithesishq/bombadil/releases/download/v%version%/bombadil\-aarch64\-darwin chmod +x bombadil .EE .PP Put the binary somewhere on your \f[CR]PATH\f[R], like in \f[CR]\(ti/.local/bin\f[R] if that is configured. .IP .EX mv ./bombadil \(ti/.local/bin/bombadil .EE .PP You should now be able to run it: .IP .EX bombadil \-\-version .EE .PP Do not download the executable with your web browser. It will be blocked by GateKeeper. Linux .PP Download the \f[CR]bombadil\f[R] binary and make it executable: .IP .EX curl \-L \-o bombadil https://github.com/antithesishq/bombadil/releases/download/v%version%/bombadil\-x86_64\-linux chmod +x bombadil .EE .PP Put the binary somewhere on your \f[CR]PATH\f[R], like in \f[CR]\(ti/.local/bin\f[R] if that is configured. .IP .EX mv ./bombadil \(ti/.local/bin/bombadil .EE .PP You should now be able to run it: .IP .EX bombadil \-\-version .EE Nix (flake) .IP .EX nix run github:antithesishq/bombadil .EE GitHub Actions .IP .EX \f[B]\-\f[R] uses\f[B]:\f[R] antithesishq/bombadil\-action\(atv2 with\f[B]:\f[R] driver\f[B]:\f[R] browser origin\f[B]:\f[R] https://your\-app.example.com specification\f[B]:\f[R] ./bombadil/specification.ts time\-limit\f[B]:\f[R] 5m exit\-on\-violation\f[B]:\f[R] true output\-path\f[B]:\f[R] bombadil\-output .EE .PP See \c .UR https://github.com/antithesishq/bombadil-action its README .UE \c \ for detailed instructions. .PP Docker images are not yet available, but coming soon. .PP If you want to compile from source, see \c .UR https://github.com/antithesishq/bombadil/tree/main/docs/development/contributing.md Contributing .UE \c \&. .SS TypeScript support When writing specifications in TypeScript, you\(cqll want the types available. If you installed Bombadil via npm, you already have them \(em skip ahead to Your first test. .PP Otherwise, install the package with your package manager of choice: npm .IP .EX npm install \-\-save\-dev \(atantithesishq/bombadil .EE Yarn .IP .EX yarn add \-\-dev \(atantithesishq/bombadil .EE Bun .IP .EX bun add \-\-development \(atantithesishq/bombadil .EE .PP Or use the files provided in the release package. .SS Your first test With the CLI installed, let\(cqs run a browser test just to see that things are working: .IP .EX bombadil browser test https://en.wikipedia.org \-\-output\-path my\-test .EE .PP This will run until you shut it down using Ctrl+C. Any property violations will be logged as errors, and with the \f[CR]\-\-output\-path\f[R] option you will get results to inspect afterwards. .PP Launch the \f[I]Bombadil Inspect\f[R] tool to see what happened in the test you just ran: .IP .EX bombadil browser inspect my\-test .EE .PP This will open a web application in your browser, which has some features to highlight: .IP \(bu 2 This interface is focused on \f[I]state transitions\f[R], i.e.\ the state before and after each action. .IP \(bu 2 On the left is the actions list, which you can use to navigate the state transitions by clicking the actions. .IP \(bu 2 In the bottom you\(cqll see the timeline, which you can scrub (click and drag) with your mouse. The timeline also shows the currently selected state transition. .IP \(bu 2 If there were any violations found in the test, they\(cqll be shown as exclamation mark icons in the timeline. .PP No violations? That\(cqs fine, Wikipedia is pretty solid! This confirms that Bombadil runs and produces results. .SS Reproducing violations If Bombadil finds a bug in a project you\(cqre working on, you might want to reproduce that test case when working on a bug fix. That is, have Bombadil perform the same sequence of actions to reach the same state. .PP Use the \f[CR]\-\-reproduce\f[R] option and point it to the output directory of the original test run: .IP .EX bombadil browser test \-\-reproduce=my\-test http://example.com .EE .PP Reproductions are not guaranteed to succeed; if they diverge, Bombadil fails with an error. For reproductions to succeed, it\(cqs important to use the same options as in the original test. After a test run, Bombadil prints both the \f[CR]inspect\f[R] and \f[CR]\-\-reproduce\f[R] commands you can use. .SH Specification language To extend Bombadil with domain\-specific knowledge, you write specifications. These are plain TypeScript or JavaScript \c .UR https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules modules .UE \c \ that use the library provided by Bombadil to export \f[I]properties\f[R] and \f[I]action generators\f[R]. .PP Here\(cqs how you run Bombadil with a custom specification: .IP .EX bombadil browser test https://example.com example.ts .EE .PP For a full listing of CLI options, see the reference. .SS Structure A specification is a regular ES module. The following examples use TypeScript, but you may also write them in JavaScript. If you do use TypeScript, you\(cqll want to install the types from \(atantithesishq/bombadil. .PP Both properties and action generators are exposed to Bombadil as exports: .IP .EX \f[B]export\f[R] \f[B]const\f[R] myProperty = ...; \f[B]export\f[R] \f[B]const\f[R] myAction = ...; .EE .PP You may split up your specification into multiple modules and structure it the way you like, but the top\-level specification you give to Bombadil must only export properties and action generators. .SS Importing modules and files You can split your specification up into multiple modules and import them from the top level specification module: .IP .EX \f[B]import\f[R] { thing } \f[B]from\f[R] \(dq./lib/thing.ts\(dq; .EE .PP If you install packages from NPM or similar, you may import those too: .IP .EX \f[B]import\f[R] equal \f[B]from\f[R] \(dqdeep\-equal\(dq; .EE .PP The runtime is limited, so many existing packages might not work, e.g.\ if they import NodeJS packages. .SS Non\-code files You can import JSON, text, or binary data. This is useful for bootstrapping tests with reference data, dictionaries, configuration, etc. Use ES import attributes to specify the type of import: .IP .EX \f[B]import\f[R] data \f[B]from\f[R] \(dq./fixtures/data.json\(dq \f[B]with\f[R] { type: \(dqjson\(dq }; \f[B]import\f[R] contents \f[B]from\f[R] \(dq./wordlist.txt\(dq \f[B]with\f[R] { type: \(dqtext\(dq }; \f[B]import\f[R] raw \f[B]from\f[R] \(dq./snapshot.dat\(dq \f[B]with\f[R] { type: \(dqbinary\(dq }; .EE .PP When using \f[CR]\(dqtext\(dq\f[R] you get a \f[CR]string\f[R], and when using \f[CR]\(dqbinary\(dq\f[R] you get \f[CR]Uint8Array\f[R]. .PP For JSON data, you can also just use the file extension: .IP .EX \f[B]import\f[R] data \f[B]from\f[R] \(dq./fixtures/data.json\(dq; .EE .SS Default properties and action generators Bombadil comes with a set of default properties and action generators that work for most web applications. You\(cqll probably want to reexport all or at least most of these: .IP .EX \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; .EE .PP In fact, these defaults are exactly what are used when running tests without a custom specification file. If you want to selectively pick just a subset of these, replace the \f[CR]*\f[R] with the relevant names: .IP .EX \f[B]export\f[R] { \f[I]// Properties\f[R] noUncaughtExceptions, \f[I]// Actions\f[R] clicks, reload, } \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; .EE .PP You may freely combine defaults with your own properties and action generators. All properties and action generators exported by the top\-level module are used by Bombadil. .PP The browser defaults include properties checking for uncaught exceptions, unhandled promise rejections, error logs, HTTP 4xx and 5xx responses, and more. On the actions side, there are generators for general navigation and interaction with semantic HTML elements. .SS Language features The specification language of Bombadil, embedded in TypeScript or JavaScript, has a small set of central concepts. This section describes them in detail. .SS Properties A property is a description of how the system under test should behave \f[I]in general\f[R]. This is different from example\-based testing (e.g.\ Playwright, Cypress, or Selenium) where you describe how it behaves for \f[I]particular\f[R] cases. .PP The most intuitive kind of property, which you might have come across before, is an \f[I]invariant\f[R]: a condition that should always be true. In Bombadil, invariants are expressed using the \f[CR]always\f[R] temporal operator: .IP .EX always( \f[I]// some condition that should always be true\f[R] ) .EE .PP To instruct Bombadil to check your property, you must export it from your specification module. Its name is used in error reports, so give the export a meaningful name. .IP .EX \f[B]export\f[R] \f[B]const\f[R] hasTitle = always( \f[I]// check that there\(aqs a title rendered somehow\f[R] ); .EE .PP You may export multiple properties, including the defaults, and they\(cqll all be checked independently. But how do you \(lqcheck that there\(cqs a title somehow\(rq? You need access to the browser, and for that, you use \f[I]extractors\f[R]. .SS Extractors In order to describe a condition about the web page you\(cqre testing, you first need to extract state. This is done with the \f[CR]extract\f[R] function, which runs inside the browser on every state that Bombadil decides to capture. .IP .EX extract(state \f[B]=>\f[R] ...) .EE .PP You give it a function that takes the current browser state as an argument, and returns JSON\-serializable data. The state object contains a bunch of things, but the most important are \f[CR]document\f[R] and \f[CR]window\f[R] \(em the same ones you have access to in JavaScript running in a browser. .PP To extract the title, you\(cqd define this at the top level of your specification: .IP .EX \f[B]const\f[R] title = extract(state \f[B]=>\f[R] state.document.title || \(dq\(dq); .EE .PP The \f[CR]title\f[R] value is not a \f[CR]string\f[R] though \(em it\(cqs a \f[CR]Cell\f[R], a stateful value that changes over time. For every new state captured by Bombadil, the extractor function gets run, and the cell is updated with its return value. .PP Using the \f[CR]title\f[R] cell, you can define the property: .IP .EX \f[B]export\f[R] \f[B]const\f[R] hasTitle = always(() \f[B]=>\f[R] title.current !== \(dq\(dq ); .EE .PP Two things to note about this example: .IP "1." 3 The expression passed to \f[CR]always\f[R] is a function that takes no arguments \(em a \f[I]thunk\f[R]. This is because it needs to be evaluated in every state. It needs to \f[I]always\f[R] be true, not just once, and that\(cqs why you need to supply the thunk rather than a \f[CR]boolean\f[R]. .IP "2." 3 To get the \f[CR]string\f[R] value out of the cell, you use \f[CR].current\f[R]. .PP This is a custom property using the \f[I]temporal\f[R] operator called \f[CR]always\f[R]. There are other temporal operators, described in Formulas below. .SS Formulas Formulas and temporal operators may sound scary, but fear not \(em they are essentially ways of expressing \(lqconditions over time\(rq. Here are some quick facts about formulas and temporal operators: .IP \(bu 2 Temporal operators return formulas. .IP \(bu 2 Every property in Bombadil is a formula (of the \f[CR]Formula\f[R] type). .IP \(bu 2 A temporal operator is a function that takes some subformula and evaluates it over time. .IP \(bu 2 Different temporal operators evaluate their subformulas in different ways. .IP \(bu 2 Bombadil evaluates formulas against a sequence of states to check if they \f[I]hold true\f[R]. .PP Temporal operator types include \f[CR]always\f[R], as discussed in the example in Extractors above, and also \f[CR]eventually\f[R] and \f[CR]next\f[R]. Here\(cqs an informal[3] description of how they work: .IP \(bu 2 \f[CR]always(x)\f[R] holds if \f[CR]x\f[R] holds in \f[I]this\f[R] and \f[I]every future\f[R] state .IP \(bu 2 \f[CR]next(x)\f[R] holds if \f[CR]x\f[R] holds in \f[I]the next\f[R] state .IP \(bu 2 \f[CR]eventually(x)\f[R] holds if \f[CR]x\f[R] holds in \f[I]this\f[R] or \f[I]any future\f[R] state .PP They accept \f[I]subformulas\f[R] as arguments. You\(cqll notice in the example with \f[CR]always\f[R] above, the argument was a thunk. This still works, because the operators automatically convert thunks into formulas. In fact, there\(cqs an operator for doing that explicitly, called \f[CR]now\f[R]: .IP .EX always(now(() \f[B]=>\f[R] title.current !== \(dq\(dq)) .EE .PP You normally don\(cqt have to use the \f[CR]now\f[R] operator, unless you want to use \f[I]logical connectives\f[R] at the formula level. They are defined as methods on formulas: .IP \(bu 2 \f[CR]x.and(y)\f[R] holds if \f[CR]x\f[R] holds and \f[CR]y\f[R] holds .IP \(bu 2 \f[CR]x.or(y)\f[R] holds if \f[CR]x\f[R] holds or \f[CR]y\f[R] holds .IP \(bu 2 \f[CR]x.implies(y)\f[R] holds if \f[CR]x\f[R] doesn\(cqt hold or \f[CR]y\f[R] holds .PP There\(cqs also negation, both as a function and as a method on formulas, i.e.\ \f[CR]not(x)\f[R] and \f[CR]x.not()\f[R]. .PP The \f[CR]now\f[R] operator is useful when expressing single\-state preconditions. The following property checks that pressing a button shows a spinner that is eventually hidden again: .IP .EX \f[B]const\f[R] buttonPressed = extract(() \f[B]=>\f[R] ...); \f[B]const\f[R] spinnerVisible = extract(() \f[B]=>\f[R] ...); now(() \f[B]=>\f[R] buttonPressed.current).implies( now(() \f[B]=>\f[R] spinnerVisible.current) .and(eventually(() \f[B]=>\f[R] !spinnerVisible.current)) ) .EE .PP You can build more advanced formulas, and even include nested temporal operators, but the basics are often powerful enough. See the examples at the bottom for more inspiration. .SS Action generators In addition to exporting properties in a specification, you export action generators. A generator is an object with a \f[CR]generate()\f[R] method. An action generator generates values of type \f[CR]Tree\f[R]. .PP Like with default properties, there are default actions provided by Bombadil. These will get you a long way, but there are times where you\(cqll need to define your own action generators. .PP For every state that Bombadil captures, all action generators are run, contributing to a tree structure of \f[I]possible\f[R] actions. Bombadil then randomly picks one in that tree. Why a tree, though? It\(cqs because the branches are \f[I]weighted\f[R] \(em equally, by default. But you can override this to control the probability of an action being picked. .PP To define a custom action generator, you use the \f[CR]actions\f[R] function, which takes a thunk that returns an array of actions: .IP .EX \f[B]export\f[R] \f[B]const\f[R] myAction = actions(() \f[B]=>\f[R] { \f[B]return\f[R] [ ... ]; }); .EE .PP In the returned array, each element is a value of the following \f[CR]Action\f[R] type, provided by the NPM package: .IP .EX \f[B]interface\f[R] Point { x: number; y: number; } \f[B]type\f[R] Action = | \(dqBack\(dq | \(dqForward\(dq | \(dqReload\(dq | \(dqWait\(dq | { Click: { name: string; content?: string; point: Point } } \f[I]// Many others...\f[R] ; .EE .PP Here\(cqs a generator for clicks in the center of a \f[CR]canvas\f[R] element: .IP .EX \f[B]const\f[R] canvasCenter = extract((state) \f[B]=>\f[R] { \f[B]const\f[R] canvas = state.document.querySelector(\(dq#my\-canvas\(dq); \f[B]if\f[R] (!canvas) { \f[B]return\f[R] \f[B]null\f[R]; } \f[B]const\f[R] rect = canvas.getBoundingClientRect(); \f[B]if\f[R] (rect.width > 0 && rect.height > 0) { \f[B]return\f[R] { x: rect.left + rect.width / 2, y: rect.top + rect.height / 2, }; } \f[B]return\f[R] \f[B]null\f[R]; }); \f[B]export\f[R] \f[B]const\f[R] clickCanvas = actions(() \f[B]=>\f[R] { \f[B]const\f[R] point = canvasCenter.current; \f[B]return\f[R] point ? [{ Click: { name: \(dqcanvas\(dq, point } }] : []; }); .EE .PP For double\-click actions, specify the delay between clicks in milliseconds (0\-1000ms): .IP .EX \f[B]export\f[R] \f[B]const\f[R] doubleClickCanvas = actions(() \f[B]=>\f[R] { \f[B]const\f[R] point = canvasCenter.current; \f[B]return\f[R] point ? [{ DoubleClick: { name: \(dqcanvas\(dq, point, delayMillis: 100, } }] : []; }); .EE .PP The actions you return must be possible to perform in the current state. Your action generators should therefore depend on cells and validate your actions before returning them, as done with \f[CR]canvasCenter\f[R] in the previous example. Another example is the \f[CR]back\f[R] action generator provided by Bombadil, which checks that there\(cqs a history entry to go back to, otherwise returning \f[CR][]\f[R]. .PP To give actions different weights, use the \f[CR]weighted\f[R] combinator and wrap each subgenerator in an array with the weight as the first element: .IP .EX \f[B]export\f[R] \f[B]const\f[R] navigation = weighted([ [10, back], [1, forward], [1, reload], ]); .EE .SS Examples These are full, runnable examples of properties and action generators you might need in your own testing with Bombadil. Think of them as design patterns for properties. Each example is a self\-contained specification file. .SS Invariant: max notification count This is a simple property checking that there are never more than five notifications shown. .IP .EX \f[B]import\f[R] { extract, always } \f[B]from\f[R] \(dq\(atantithesishq/bombadil\(dq; \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; \f[B]const\f[R] notificationCount = extract((state) \f[B]=>\f[R] state.document.body.querySelectorAll(\(dq.notification\(dq).length, ); \f[B]export\f[R] \f[B]const\f[R] max_notifications_shown = always(() \f[B]=>\f[R] notificationCount.current <= 5, ); .EE .SS Sliding window: constant notification count This property checks that the notification count doesn\(cqt change \(em that it is the same as in the first state. Note how this property evaluates \f[CR]notificationCount.current\f[R] in the outer thunk, and then uses that in the inner thunk to compare against the current value. .IP .EX \f[B]import\f[R] { extract, always, now, time } \f[B]from\f[R] \(dq\(atantithesishq/bombadil\(dq; \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; \f[B]const\f[R] notificationCount = extract((state) \f[B]=>\f[R] state.document.body.querySelectorAll(\(dq.notification\(dq).length, ); \f[B]export\f[R] \f[B]const\f[R] constantNotificationCount = now(() \f[B]=>\f[R] { \f[B]const\f[R] initial = notificationCount.current; \f[B]return\f[R] always(() \f[B]=>\f[R] notificationCount.current === initial, ); }); .EE .SS Guarantee: error disappears A \f[I]guarantee property\f[R] checks that something good eventually happens, within some time bound. Here is a property that checks that error messages disappear within five seconds. .IP .EX \f[B]import\f[R] { extract, always, now, eventually } \f[B]from\f[R] \(dq\(atantithesishq/bombadil\(dq; \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; \f[B]const\f[R] errorMessage = extract((state) \f[B]=>\f[R] state.document.body.querySelector(\(dq.error\(dq)?.textContent ?? \f[B]null\f[R], ); \f[B]export\f[R] \f[B]const\f[R] errorDisappears = always( now(() \f[B]=>\f[R] errorMessage.current !== \f[B]null\f[R]).implies( eventually(() \f[B]=>\f[R] errorMessage.current === \f[B]null\f[R]) .within(5, \(dqseconds\(dq), ), ); .EE .SS Contextful guarantee: notification includes past value This property checks that if there\(cqs a non\-blank name entered, and it is submitted, then eventually there will be a notification that includes the name. This example uses an outer thunk to force a cell value (\f[CR]nameEntered\f[R]) at every state, and then closes over that value with the inner thunk passed to \f[CR]eventually\f[R]. .IP .EX \f[B]import\f[R] { extract, always, now, next, eventually } \f[B]from\f[R] \(dq\(atantithesishq/bombadil\(dq; \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; \f[B]const\f[R] name = extract((state) \f[B]=>\f[R] { \f[B]const\f[R] element = state.document.body.querySelector(\(dq#name\-field\(dq); \f[B]return\f[R] (element \f[B]as\f[R] HTMLInputElement | null)?.value ?? \f[B]null\f[R]; }); \f[B]const\f[R] submitInProgress = extract((state) \f[B]=>\f[R] state.document.body.querySelector(\(dqsubmit.progress\(dq) !== \f[B]null\f[R], ); \f[B]const\f[R] notificationText = extract((state) \f[B]=>\f[R] state.document.body.querySelector(\(dq.notification\(dq)?.textContent ?? \f[B]null\f[R], ); \f[B]export\f[R] \f[B]const\f[R] notificationIncludesMessage = always(() \f[B]=>\f[R] { \f[B]const\f[R] nameEntered = name.current?.trim() ?? \(dq\(dq; \f[B]return\f[R] now(() \f[B]=>\f[R] nameEntered !== \(dq\(dq) .and(next(() \f[B]=>\f[R] submitInProgress.current)) .implies(eventually(() \f[B]=>\f[R] notificationText.current?.includes(nameEntered) ?? \f[B]false\f[R], ).within(5, \(dqseconds\(dq)); }); .EE .SS State machine: counter This property models a counter as a state machine, checking that the counter only transitions by staying the same, incrementing by 1, or decrementing by 1 (no invalid jumps allowed). .IP .EX \f[B]import\f[R] { extract, always, now, next } \f[B]from\f[R] \(dq\(atantithesishq/bombadil\(dq; \f[B]export\f[R] * \f[B]from\f[R] \(dq\(atantithesishq/bombadil/browser/defaults\(dq; \f[B]const\f[R] counterValue = extract((state) \f[B]=>\f[R] { \f[B]const\f[R] element = state.document.body.querySelector(\(dq#counter\(dq); \f[B]return\f[R] parseInt(element?.textContent ?? \(dq0\(dq, 10); }); \f[B]const\f[R] unchanged = now(() \f[B]=>\f[R] { \f[B]const\f[R] current = counterValue.current; \f[B]return\f[R] next(() \f[B]=>\f[R] counterValue.current === current); }); \f[B]const\f[R] increment = now(() \f[B]=>\f[R] { \f[B]const\f[R] current = counterValue.current; \f[B]return\f[R] next(() \f[B]=>\f[R] counterValue.current === current + 1); }); \f[B]const\f[R] decrement = now(() \f[B]=>\f[R] { \f[B]const\f[R] current = counterValue.current; \f[B]return\f[R] next(() \f[B]=>\f[R] counterValue.current === current \- 1); }); \f[B]export\f[R] \f[B]const\f[R] counterStateMachine = always(unchanged.or(increment).or(decrement)); .EE .PP If this specification exports the \f[CR]reload\f[R] action, the \f[CR]unchanged\f[R] property becomes relevant.[4] Unless this application stored the state of the counter somehow, reloading the page would clear the counter, which this property would catch as a violation. .SH Reference .SS Command\-line interface .SS Exit codes .PP .TS tab(@); r l. T{ Code T}@T{ Meaning T} _ T{ 0 T}@T{ Test completed normally (including time limit) T} T{ 1 T}@T{ Other error T} T{ 2 T}@T{ Property violation(s) detected T} .TE .SS bombadil browser test \f[CR]bombadil\f[R] \f[CR]test\f[R] \f[CR][OPTIONS]\f[R] \f[CR]\f[R] \f[CR][SPECIFICATION_FILE]\f[R] .PP .TS tab(@); l l. T{ Argument T}@T{ Description T} _ T{ \f[CR]\f[R] T}@T{ Starting URL of the test (also used as a boundary so that Bombadil doesn\(cqt navigate to other websites) T} T{ \f[CR][SPECIFICATION_FILE]\f[R] T}@T{ A custom specification in TypeScript or JavaScript, using the \f[CR]\(atantithesishq/bombadil\f[R] package on NPM T} .TE .PP .TS tab(@); l l r. T{ Option T}@T{ Description T}@T{ Default T} _ T{ \f[CR]\-\-output\-path \f[R] T}@T{ Where to store output data (trace, screenshots, etc.) T}@T{ T} T{ \f[CR]\-\-output\-path\-overwrite\f[R] T}@T{ Overwrite any existing \f[CR]trace.jsonl\f[R] at \f[CR]\-\-output\-path\f[R]. Without this flag, Bombadil refuses to write when one already exists. T}@T{ T} T{ \f[CR]\-\-exit\-on\-violation\f[R] T}@T{ Whether to exit the test when first failing property is found (useful in development and CI) T}@T{ T} T{ \f[CR]\-\-time\-limit \f[R] T}@T{ Maximum time to run the test; reaching the limit is treated as normal completion. Accepts a number with a unit suffix: s (seconds), m (minutes), h (hours), or d (days). Examples: 30s, 5m, 2h, 1d T}@T{ T} T{ \f[CR]\-\-width \f[R] T}@T{ Browser viewport width in pixels T}@T{ 1024 T} T{ \f[CR]\-\-height \f[R] T}@T{ Browser viewport height in pixels T}@T{ 768 T} T{ \f[CR]\-\-device\-scale\-factor \f[R] T}@T{ Scaling factor of the browser viewport, mostly useful on high\-DPI monitors when in headed mode T}@T{ 2 T} T{ \f[CR]\-\-instrument\-javascript \f[R] T}@T{ What types of JavaScript to instrument for coverage tracking. Comma\-separated list of: \(lqfiles\(rq, \(lqinline\(rq T}@T{ files,inline T} T{ \f[CR]\-\-chrome\-grant\-permissions \f[R] T}@T{ Comma\-separated list of Chrome permissions to grant. Examples: local\-network\-access, geolocation, notifications. T}@T{ local\-network\-access,local\-network,loopback\-network T} T{ \f[CR]\-\-header \f[R] T}@T{ Extra HTTP header to send with all browser requests, in \f[CR]KEY=VALUE format\f[R]. Can be specified multiple times. T}@T{ T} T{ \f[CR]\-\-reproduce \f[R] T}@T{ Reproduce a previous test run from a trace file, instead of random exploration. Mutually exclusive with \f[CR]\-\-time\-limit\f[R] and \f[CR]\-\-exit\-on\-violation\f[R]. T}@T{ T} T{ \f[CR]\-\-headless\f[R] T}@T{ Whether the browser should run in a visible window or not T}@T{ T} T{ \f[CR]\-\-no\-sandbox\f[R] T}@T{ Disable Chromium sandboxing T}@T{ T} T{ \f[CR]\-h, \-\-help\f[R] T}@T{ Print help T}@T{ T} .TE .SS bombadil browser test\-external \f[CR]bombadil\f[R] \f[CR]test\-external\f[R] \f[CR][OPTIONS]\f[R] \f[CR]\f[R] \f[CR][SPECIFICATION_FILE]\f[R] .PP .TS tab(@); l l. T{ Argument T}@T{ Description T} _ T{ \f[CR]\f[R] T}@T{ Starting URL of the test (also used as a boundary so that Bombadil doesn\(cqt navigate to other websites) T} T{ \f[CR][SPECIFICATION_FILE]\f[R] T}@T{ A custom specification in TypeScript or JavaScript, using the \f[CR]\(atantithesishq/bombadil\f[R] package on NPM T} .TE .PP .TS tab(@); l l r. T{ Option T}@T{ Description T}@T{ Default T} _ T{ \f[CR]\-\-output\-path \f[R] T}@T{ Where to store output data (trace, screenshots, etc.) T}@T{ T} T{ \f[CR]\-\-output\-path\-overwrite\f[R] T}@T{ Overwrite any existing \f[CR]trace.jsonl\f[R] at \f[CR]\-\-output\-path\f[R]. Without this flag, Bombadil refuses to write when one already exists. T}@T{ T} T{ \f[CR]\-\-exit\-on\-violation\f[R] T}@T{ Whether to exit the test when first failing property is found (useful in development and CI) T}@T{ T} T{ \f[CR]\-\-time\-limit \f[R] T}@T{ Maximum time to run the test; reaching the limit is treated as normal completion. Accepts a number with a unit suffix: s (seconds), m (minutes), h (hours), or d (days). Examples: 30s, 5m, 2h, 1d T}@T{ T} T{ \f[CR]\-\-width \f[R] T}@T{ Browser viewport width in pixels T}@T{ 1024 T} T{ \f[CR]\-\-height \f[R] T}@T{ Browser viewport height in pixels T}@T{ 768 T} T{ \f[CR]\-\-device\-scale\-factor \f[R] T}@T{ Scaling factor of the browser viewport, mostly useful on high\-DPI monitors when in headed mode T}@T{ 2 T} T{ \f[CR]\-\-instrument\-javascript \f[R] T}@T{ What types of JavaScript to instrument for coverage tracking. Comma\-separated list of: \(lqfiles\(rq, \(lqinline\(rq T}@T{ files,inline T} T{ \f[CR]\-\-chrome\-grant\-permissions \f[R] T}@T{ Comma\-separated list of Chrome permissions to grant. Examples: local\-network\-access, geolocation, notifications. T}@T{ local\-network\-access,local\-network,loopback\-network T} T{ \f[CR]\-\-header \f[R] T}@T{ Extra HTTP header to send with all browser requests, in \f[CR]KEY=VALUE format\f[R]. Can be specified multiple times. T}@T{ T} T{ \f[CR]\-\-reproduce \f[R] T}@T{ Reproduce a previous test run from a trace file, instead of random exploration. Mutually exclusive with \f[CR]\-\-time\-limit\f[R] and \f[CR]\-\-exit\-on\-violation\f[R]. T}@T{ T} T{ \f[CR]\-\-remote\-debugger \f[R] T}@T{ Address to the remote debugger\(cqs server, e.g.\ http://localhost:9222 T}@T{ T} T{ \f[CR]\-\-create\-target\f[R] T}@T{ Whether Bombadil should create a new tab and navigate to the origin URL in it, as part of starting the test (this should probably be false if you test an Electron app) T}@T{ T} T{ \f[CR]\-h, \-\-help\f[R] T}@T{ Print help T}@T{ T} .TE .SS bombadil browser inspect \f[CR]bombadil\f[R] \f[CR]inspect\f[R] \f[CR][OPTIONS]\f[R] \f[CR]\f[R] .PP .TS tab(@); l l. T{ Argument T}@T{ Description T} _ T{ \f[CR]\f[R] T}@T{ Path to trace.jsonl file or directory containing it T} .TE .PP .TS tab(@); l l r. T{ Option T}@T{ Description T}@T{ Default T} _ T{ \f[CR]\-\-port \f[R] T}@T{ Port to bind the inspect server to T}@T{ 1073 T} T{ \f[CR]\-\-no\-open\f[R] T}@T{ Skip auto\-opening browser T}@T{ T} T{ \f[CR]\-h, \-\-help\f[R] T}@T{ Print help T}@T{ T} .TE .SH NOTES .SS [1] .PP See the \c .UR https://antithesis.com/docs/resources/property_based_testing/ property based testing .UE \c \ guide for an introduction. .SS [2] .PP You can also configure Bombadil to exit on the first found violation. .SS [3] .PP Formally, the properties in Bombadil use a flavor of \c .UR https://en.wikipedia.org/wiki/Linear_temporal_logic Linear Temporal Logic .UE \c , if you\(cqre into dense theoretical stuff. .SS [4] .PP A state transition that allows for nothing to change is a way of making a property \(lqstutter\-invariant\(rq, as it\(cqs called in the literature.