Betwixt Representation

library(betwixt)

Introduction

The design of Betwixt starts from the principles of tidy data introduced by Wickham and implemented throughout the tidyverse. Tidy data provides a simple and powerful relational discipline: variables are represented by columns, observations by rows, and different types of observational units by tables. This organisation makes data transformation, visualisation and modelling easier to express as reproducible computational operations.

The dataset package builds on these principles for statistical data that must remain interpretable beyond the computational environment in which they were originally prepared. Its semantic dataset representation makes variable definitions and other metadata explicit, and its data-cube perspective distinguishes the dimensions, measures and attributes associated with statistical observations. This is particularly important when datasets originate from different sources, waves or computational environments: two columns can have identical names and compatible storage types without necessarily representing the same statistical concept.

Betwixt addresses a related problem at the boundary between semantic knowledge production and review. Candidate knowledge may be produced by different analysts, software systems, AI agents or computational environments and subsequently presented for review in another environment. Some of the semantic decisions required to combine these outputs cannot safely be reduced to syntactic operations such as matching column names or identifiers. They may require an explicit judgement about observational-unit identity, variable meaning, or the relationship between a supplied resource and the subject of an assertion.

The Betwixt representation therefore retains the basic discipline of tidy data while introducing a small number of deliberate extensions for semantic review. Serialised input objects establish the boundary between producing and reviewing environments; context_* attributes carry information needed to interpret a review unit without themselves becoming objects of review; candidate and reviewed states make semantic transitions explicit; and stable coordinates identify review units and their atomic assertions across successive states.

Betwixt is generally neutral with respect to the domain semantics of an assertion. A subject, predicate, or value may draw on CIDOC CRM, DCTERMS, Schema.org, a local vocabulary, or no formal ontology at all. Betwixt nevertheless defines the semantics required for review itself. In particular, it distinguishes review outcomes such as corroboration, deferral and rejection, and permits a small number of distinguished values in an otherwise open value range. These include an unknown applicable value and the absence of any possible value in a particular state. Such values express semantic information rather than computational missingness; their projection into a particular domain representation remains governed by the target vocabulary and its constraints.

This vignette explains these representational choices and their relationship to tidy data, data cubes and simple graph representations. Its purpose is not to introduce an alternative general-purpose data model. Rather, it describes a bounded, representation-independent intermediate representation intended to make semantic knowledge explicit, inspectable, reviewable and reproducible when it crosses computational boundaries.

From evidence to input

Earlier versions of Betwixt used terms such as input_url, input_media_url, and input_description. These names made a semantic commitment that is not always warranted. A resource supplied to a review process is not necessarily evidence for the assertion being reviewed. It may be a candidate dataset, source document, image, previous semantic state, generated output, or another artefact whose role has yet to be established.

Betwixt therefore uses input as the more neutral term.

An input is a serialised artefact supplied to a Betwixt review process. Examples include:

candidate.csv
candidate.xlsx
eurobarometer794.sav
https://example.com/delini001.jpg
archive.wacz

input describes the artefact’s operational role in the review process rather than asserting what kind of thing it is or what evidential role it has.

This also establishes an important boundary. Betwixt does not need a special input such as [current environment]. Candidate production may take place programmatically in an R, Python, AI-agent, or other computational environment, but Betwixt begins after a serialisation boundary:

[current computational environment]
        ↓
candidate production
        ↓
serialisation
        ↓
candidate.csv
        ↓
read_to_betwixt()
        ↓
Betwixt review

The serialised candidate is therefore a reproducible handover object between knowledge production and review.

Computational-environment boundaries

The conventional tidy-data workflow normally takes place within a single analytical environment. An analyst reads or constructs data, understands what its variables and observational units mean, tidies them, and applies deterministic transformations.

A great deal of semantic knowledge can consequently remain implicit. If an analyst creates a variable called CPI, she may know whether it means Consumer Price Index or Corruption Perceptions Index. That knowledge need not be represented computationally for her own pipeline to execute.

This assumption becomes unsafe when data cross computational or organisational boundaries.

Betwixt is designed for situations in which:

• candidates may originate in multiple computational environments;
• different analysts or agents may produce them;
• inputs may have different modalities and serialisation formats;
• candidate production and candidate review need not use the same software;
• human judgement may be required before a deterministic transformation can continue.

For example:

AI agent A ───────→ serialised candidate A ──┐
AI agent B ───────→ serialised candidate B ──┤
Python pipeline ──→ serialised candidate C ──┤
R workflow ───────→ serialised candidate D ──┤
Excel user ───────→ serialised candidate E ──┘
                                             ↓
                                  Betwixt review environment
                                             ↓
                                   stabilised semantic state

The review always takes place in a particular computational environment, but the reviewed inputs need not have originated there.

This is particularly important for human review. Tidyverse-style pipelines are deterministic once their transformations have been specified. Semantic preparation, however, often contains an exploratory phase in which a human determines what observations mean, whether two variables correspond, or whether two identifiers refer sufficiently to the same thing.

Betwixt brings this otherwise implicit semantic labour into the reproducible workflow.

Semantic roles

Betwixt defines semantic roles independently of their computational representation. The same atomic semantic assertion can therefore be represented in tabular, graph, or tensor form without making any of these representations canonical.

Betwixt role	Tabular representation	RDF representation	Tensor representation
subject	Value in the subject column	RDF subject position	Index on the subject/entity axis
predicate	Wide column name, or value in predicate	RDF predicate position	Index on the relation/predicate axis
value	Cell value, or value in value	RDF object position	Index/value on the object/value axis

In this sense, subject, predicate, and value identify semantic roles rather than storage types. Their concrete realisation depends on the computational representation. A value occupying the subject role in a Betwixt table, for example, need not already be encoded as a valid RDF subject; it may subsequently be realised as one when the assertion is projected to RDF.

The Betwixt tuple

Betwixt uses a specialised tuple as its intermediate representation for semantic review. A Betwixt tuple is a review unit that associates a subject with one or more candidate assertions and their reviewed values. It may additionally associate the review unit with an input that informs the review, contextual attributes, and reviewer feedback.

In a tabular representation, a Betwixt tuple is represented by a row and identified by row_id. The tuple is nevertheless a semantic construct rather than a physical row position: it retains its identity when a table is sorted, filtered, reshaped, serialised, or reconstructed.

The tuple therefore introduces a level of organisation above the atomic assertion:

Betwixt construct	Role
subject	Semantic entity about which assertions are made
input	Serialised artefact supplied to the review process
Betwixt tuple	Review unit connecting the subject, candidate and reviewed assertions, and optionally input, context, and reviewer feedback
assertion	Atomic subject–predicate–value semantic claim represented within the review unit

A single Betwixt tuple may contain several atomic assertions. Conversely, an assertion belongs to an identified review unit and can therefore be addressed independently of its physical position in a particular serialisation.

Distinguished values

The value role is generally open and domain-neutral. Betwixt does not normally interpret a value or determine whether it conforms to the ontology, vocabulary or application profile used by a particular domain representation.

Betwixt nevertheless defines a small number of distinguished values that have a specified meaning within the review representation:

Betwixt value	Meaning
btx:ValueUnknown	A value is applicable in this state, but its value is not known.
btx:NoValuePossible	No value is applicable in this state.

These values are semantically different from a missing value such as NA. A missing value does not by itself state why a value is absent. By contrast, the distinguished Betwixt values make explicit information that may otherwise remain implicit in the review process.

For example:

subject | predicate   | value
S1      | located_in  | btx:ValueUnknown
S2      | located_in  | btx:NoValuePossible

The first assertion states that located_in has an applicable but currently unknown value for S1. The second states that no located_in value is applicable to S2 in that semantic state.

These are state-relative assertions. The state is determined by the assertion’s Betwixt coordinates rather than encoded in the distinguished value itself. A later state may therefore replace either distinguished value with an ordinary domain value without contradiction.

The distinguished values also extend the range available during review. A reviewer may select:

Other... (new candidate value)
Value unknown (in this state)
No value possible (in this state)

Other... is an interface operation that introduces a new candidate value. Value unknown and No value possible, by contrast, correspond to distinguished values that can themselves occur in a reviewed semantic state and consequently become candidates in a subsequent review state.

Their Betwixt semantics do not determine how they must be represented in a downstream domain model. A direct RDF projection may use a distinguished Betwixt resource as an object, while a projection governed by CIDOC CRM, DCTERMS, SHACL constraints or another application profile may transform, omit or otherwise represent the same information according to the requirements of that model.

Betwixt is therefore ontology-neutral with respect to ordinary domain values, while assigning defined semantics to a small controlled set of values required by the review process.

Wide and dual-wide representations

A conventional wide table can be represented schematically as:

subject | p1 | p2 | p3 | context_*

The subject identifies the observational unit. Columns p1, p2, and p3 represent variables or properties concerning that subject.

For example:

subject | name | location | architect | context_source
S1      | ...  | L1       | P1        | ...

Semantically, this corresponds approximately to a star:

             name
              ↑
location ←── S1 ──→ architect

Betwixt’s dual-wide representation introduces candidate and final states so that proposed and reviewed values can be inspected together.

It may additionally place the serialised input beside the assertion:

input | input_predicate | subject | candidate_p1 | final_p1 | ... | context_*

Conceptually:

input ── input_predicate ──▶ subject ── predicate ──▶ value

The input relation and domain assertion could be normalised into separate tables:

INPUT RELATION

input             predicate    subject
delini001.jpg     depicts      granary

and:

DOMAIN RELATION

subject     predicate    value
granary     part_of      farmstead

Betwixt deliberately projects these together for review:

input          input_predicate   subject   candidate_predicate   candidate_value
delini001.jpg  depicts           granary   part_of               farmstead

This is therefore a bounded denormalisation at the review boundary.

input_predicate is important because adjacency alone does not establish why the input and subject belong together. It explicitly records the semantic relationship that warrants placing the input beside observations concerning that subject.

Long and dual-long representations

The long representation makes the assertion structure explicit:

subject | predicate | value | context_*

Instead of encoding predicates in column names:

subject | location | architect | date

the predicates become observations in their own column:

subject | predicate | value
S1      | location  | L1
S1      | architect | P1
S1      | date      | 1890

This removes much of the semantic ambiguity inherent in wide column names.

The corresponding input-aware form is:

input | input_predicate | subject | predicate | value | context_*

The dual-long review projection repeats the assertion fields for candidate and final states, for example:

input
input_predicate
subject
candidate_predicate
candidate_value
final_predicate
final_value
context_*

The essential semantic chain nevertheless remains:

input → subject → value
        predicate

with input_predicate expressing the relationship between the serialised input and the subject.

Input and context

input and context_* serve related but distinct purposes.

The input identifies the serialised artefact crossing into the review environment:

input = eurobarometer794.sav

or:

input = https://example.com/delini001.jpg

context_* columns contain information derived from that input and made available to the reviewer:

input = eurobarometer794.sav

context_wave = "79.4"
context_question = "QA12"
context_label = "..."

or:

input = delini001.jpg

context_filename = "delini001.jpg"
context_collection = "Dēliņi"
context_caption = "..."

The crucial distinction is that context columns are not themselves under review. They are attributes carried into the review observation to assist interpretation.

If a contextual value itself needs to become the subject of semantic review, it should cease to be merely context_* and be represented as a candidate assertion or promoted coordinate.

Thus:

serialised input
      ↓
extraction / projection
      ↓
context_* attributes
      ↓
review observation

The input remains the reproducible artefact; context is a review-oriented projection derived from it.

Relationship to tidy-data principles

Tidy data separates variables into columns, observations into rows, and different types of observational units into tables.

Betwixt follows this logic but operates at a boundary where some of the semantic assumptions normally held implicitly by an analyst must become explicit.

There are two principal sources of ambiguity.

Observational-unit ambiguity

Two identifiers or labels do not necessarily identify the same observational unit.

A syntactically valid join can therefore be semantically invalid.

A$CPI == B$CPI

does not establish that the two variables mean the same thing.

Similarly, lexical equality between identifiers does not necessarily establish sufficient identity for joining observations from independently governed sources.

Variable ambiguity

Wide-format column names also carry implicit semantics.

subject | CPI

does not tell an external reviewer whether CPI means Consumer Price Index, Corruption Perceptions Index, or something else.

Long form makes this dimension explicit:

subject | predicate | value

Hence:

• subject makes the observational unit explicit;
• predicate makes the variable/property explicit;
• value contains what is asserted.

Betwixt’s dual-wide representation deliberately departs from strict normalisation because the unit of work is now a review observation. Information required to determine semantic identity is placed beside the candidate assertion so that the reviewer does not need to reconstruct the computational environment in which the candidate was produced.

The resulting denormalisation is therefore bounded and purposeful rather than arbitrary.

Relationship to data-cube vocabulary

The data-cube distinction between dimensions, measures and attributes provides another useful interpretation of Betwixt tables.

These roles are treated pragmatically rather than as intrinsic properties of variables.

Measures

Measures or observed values are things asserted about an observational unit.

subject | predicate | value

The value occupies the observed position of the assertion.

Attributes

Attributes accompany an observation and assist its interpretation without themselves becoming observations under review.

Betwixt context_* columns naturally correspond to this role.

context_label
context_source_page
context_note

They are carried with the review observation but excluded from the review algebra.

Dimensions

A dimension is an observational coordinate promoted to subject identity because further knowledge may need to organise around it.

For example:

country | sex   | year | value
France  | woman | 2025 | 42

France, woman, and 2025 become particularly useful as dimensions when observations need to be sliced or grouped by them, or when additional knowledge needs to attach to them:

France ── member_of ──▶ European Union
France ── capital ─────▶ Paris

Thus dimensions can be understood as promoted subjects.

The same underlying information can therefore play different pragmatic roles:

attribute
    → carried with an observation

measure
    → asserted about an observational unit

dimension
    → promoted to identified coordinate/subject
      around which observations or further knowledge can organise

These roles are not necessarily permanent. A contextual attribute can later be promoted if it becomes important as a coordinate or join point.

Relationship to simple graph representations

The long representation corresponds directly to a labelled directed edge:

subject ── predicate ──▶ value

Multiple rows produce branching:

A ──p1──▶ B
A ──p2──▶ C
A ──p3──▶ D

A value in one assertion may become a subject elsewhere:

A ──p1──▶ B
          │
          p2
          ↓
          C

represented tabularly as:

A | p1 | B
B | p2 | C

Graph depth therefore does not require increasingly wide tables. It emerges by composing bounded assertions through subject identity.

The input-aware representation adds one preceding relation:

input ── input_predicate ──▶ subject ── predicate ──▶ value

Individual review tables consequently remain structurally simple while their composition can produce considerably richer graphs.

Blank nodes provide a useful boundary case. A blank node can represent local graph structure without requiring persistent identity. If composition is required at that point, sufficient subject identity must be established. This follows directly from the rule that composition occurs through subject identity rather than requiring a separate blank-node rule.

A broader research hypothesis follows:

A useful class of semantic graphs may be decomposable into atomic assertions and reconstructed through composition on warranted subject identity, while retaining bounded tabular structures suitable for human review.

Betwixt does not currently need to establish that this covers every possible graph. Implementation across real use cases can reveal the boundaries of the representable graph class and any additional primitives that are genuinely required.

Planes and stable coordinates

Semantic knowledge production can be represented using parallel planes rather than mixing domain knowledge, review state and provenance into a single table or graph.

At minimum, two planes are important.

Domain plane

The domain plane contains the assertions themselves:

assertion_id       | subject | predicate | value
delini-002-3-1     | granary | part_of   | farmstead
delini-002-3-2     | granary | built_in  | 1890

Provenance plane

The provenance plane records how assertions and semantic states came about:

assertion_id       | activity | agent | input | ...
delini-002-3-1     | R017     | ...   | ...   | ...
delini-002-3-2     | R018     | ...   | ...   | ...

The planes remain distinct but can be connected through stable coordinates.

For Betwixt/review, the principal structural coordinates are:

table_id
table_id
table_state
row_id
assertion_id

These coordinates operate at different levels.

• table_id identifies the larger review or knowledge-production context.
• table_id identifies a particular table within that context.
• table_state identifies an ordinal state of the table. State 0 is the input state from which subsequent review states are derived.
• row_id identifies an observational row. It is assigned in state 0 and inherited unchanged by subsequent states.
• assertion_id identifies an atomic assertion derived from an assertion-bearing position within an observational row.

The distinction between row_id and the physical row number of a particular table state is essential. At state 0 they may initially coincide:

table_state = 0

row_id | subject
1      | S1
2      | S2
3      | S3
4      | S4

Suppose all assertions associated with row_id = 3 are rejected. A later state may contain:

table_state = 1

row_id | subject
1      | S1
2      | S2
4      | S4

The third physical row is now row_id = 4. It must not be renumbered to 3.

State 0 therefore establishes the stable row coordinates of the table. Filtering, sorting, accepting, rejecting or otherwise transforming later states may change which rows are present and in which order they are displayed, but these operations do not redefine row_id.

This allows the provenance plane to refer to observations or assertions that no longer occur in a later domain state.

Assertion coordinates

In a wide representation, an observational row can contain several assertions. Their values occupy cells at the intersection of a stable row and an assertion-bearing column:

row_id | subject | located_in | built_in | material
5      | S1      | Latvia     | 1890     | wood

Conceptually, these are three atomic assertions:

row 5 × statement 1
row 5 × statement 2
row 5 × statement 3

A long projection makes these assertions explicit:

row_id | subject | predicate  | value
5      | S1      | located_in | Latvia
5      | S1      | built_in   | 1890
5      | S1      | material   | wood

The representation has changed, but the underlying assertion coordinates need not change.

In wide form, the statement coordinate can be calculated from the ordered assertion-bearing columns. In long form, it can be calculated from the ordered statements belonging to the observational row. This provides an implementation-independent rule that does not depend on a particular R reshaping operation.

An assertion_id can therefore be derived from stable table, row and statement coordinates:

table_id + row_id + statement_id

For example:

delini-002-5-1
delini-002-5-2
delini-002-5-3

The assertion_id identifies the atomic assertion position rather than its physical position in a particular serialisation.

Consequently, an assertion such as:

delini-002-5-2

can remain identifiable in the provenance plane even if it is rejected during review and omitted from subsequent domain states.

A review operation can therefore be understood as a state transition over stable coordinates:

Domain S₀ ───────────────▶ Domain S₁
     │                         │
     └── Provenance activity ──┘

State transitions change what is asserted or accepted at these coordinates; they do not redefine the coordinates established by the input state.

Programmatic and human review can use the same model. The difference is how a review decision is produced, not how the resulting domain state and provenance are represented.

Semantic coordinates for slicing, grouping and joining

The stable structural coordinates above identify tables, observations and assertions through the review process. They should be distinguished from the semantic coordinates represented by dimensions within the data itself.

Dimensions establish observational coordinates along which data can be sliced, grouped or connected to further knowledge.

For example:

country | sex   | year | value
France  | woman | 2025 | 42
France  | man   | 2025 | 39
Germany | woman | 2025 | 44

country, sex and year can be used to slice or group the observations:

country = France
sex = woman
year = 2025

This is the familiar analytical role of dimensions in a data cube.

Dimensions can also have a semantic role. Treating something as a dimension promotes its values to identified observational coordinates around which further knowledge can be organised.

For example, France can become the subject of additional assertions:

France | capital   | Paris
France | member_of | European Union

The same coordinate can therefore support both analytical operations and semantic composition:

                    France
                      │
          ┌───────────┴───────────┐
          │                       │
    slice / group             join / compose
          │                       │
 observations about        further knowledge
      France                  about France

This gives dimensions a pragmatic interpretation.

A value need not be treated as a dimension merely because it is categorical. It becomes useful as a dimension when we want either to organise observations along that coordinate or to attach further knowledge at that coordinate.

This distinguishes dimensions from measures and attributes:

measure
    → something observed or asserted

attribute
    → something carried with an observation

dimension
    → an observational coordinate promoted so that
      observations or further knowledge can organise around it

Betwixt context_* columns naturally correspond to attributes. They accompany the review observation and help the reviewer interpret it, but they are not themselves under review and do not automatically become semantic join coordinates.

If an attribute later needs to participate in slicing, grouping or semantic composition, it can be promoted to a dimension. Similarly, a value occurring in one assertion may become an identified subject elsewhere when further knowledge needs to attach to it.

Joining and semantic composition

Promotion to a dimension creates a potential composition point, but it does not by itself establish that independently governed representations identify the same thing.

For example:

TABLE A

country | value
A:France | 42

and:

TABLE B

subject  | predicate | value
B:France | capital   | Paris

cannot necessarily be joined merely because both identifiers are labelled France.

The operation requires sufficient observational-unit identity to warrant the join.

This need not imply a universal ontological assertion such as:

A:France owl:sameAs B:France

In bounded federation, a reviewer may determine only that the correspondence is sufficient for the particular operation being performed.

The progression is therefore:

value
  ↓
promoted observational coordinate
  ↓
dimension / subject identity
  ↓
potential composition point
  ↓
identity sufficient for a particular operation
  ↓
join

This connects data-cube slicing and grouping with semantic composition without treating them as the same operation.

The resulting principle is compact:

Measures terminate assertions, attributes accompany observations, and dimensions open further structure by promoting observational coordinates to subject identity.

Structural Betwixt coordinates and semantic data-cube coordinates therefore serve complementary purposes. The former preserve identity through serialisation and review states; the latter organise the knowledge represented within those states and provide potential points of semantic composition.