Information-Flow Profiles as Learning Objectives for Cooperation in Repeated Games

1Hypothesis and research design

Our central hypothesis is that the directional and temporal organization of information exchange, rather than its magnitude alone, can provide a training signal for cooperative behavior. A cooperative pair may need substantial responsiveness when establishing coordination and little new predictive information once its behavior stabilizes. An agent interacting with several kinds of partners may therefore be better characterized by its profile of sent and received information than by a single scalar average. We investigate whether these empirical profiles can guide learning while withholding cooperative-action labels, expert action targets, and payoff rewards from the optimizer.

We study repeated binary-action games before extending the idea to language-model conversations. This setting makes the random variables, action probabilities, and counterfactual next-action branches explicit. Our contribution is an auditable sequence: a behavioral grouping of fixed strategies; a directional information spectrum; a controlled loss ablation with fixed and trainable partners; direct temporal curve fits; and a reference-profile learning pilot. These are local simulations of reconstructed programs, not a reproduction of Axelrod's published numerical results. No LLM was used as a player or probability estimator in the experiments reported here.

What counts as cooperation. Two players simultaneously select C or D for \(T = 200\) rounds, observing the completed round afterward. The conventional payoffs are (3, 3) for CC, (0, 5) for CD, (5, 0) for DC, and (1, 1) for DD [Axelrod, 1980]. They define the game but are not included in our learning losses. We separately count

High individual cooperation need not imply simultaneous cooperation. These rates describe behavior against a specified opponent distribution, not an inferred intention.

2Measuring directional information

For the direction \(B \to A\), define the message \(M_t = B_t\) and the response \(U_t = A_{t+1}\). The conditioning context \(H_t\) contains the known policy rules, the horizon, both public histories through \(t-1\), and \(A_t\). It excludes \(B_t\), random seeds, RNG state, and unrevealed private random schedules. In this game the "message" is the opponent's move; there is no separate communication channel.

The policy programs provide three probabilities:

We preserve the earlier state and \(A_t\), branch the newest opponent action into C and D, update both histories consistently, and evaluate A's next policy. Fresh random choices are integrated rather than replaced by sampled actions. For Graaskamp, the evaluator averages over private schedules compatible with the public history. These disposable one-step branches do not replay an entire game or freeze its future trajectory.

The marginal and message-conditioned distributions are Bernoulli:

The observed score and its conditional expectation are

where \(h_2(p) = -p\log_2 p - (1-p)\log_2(1-p)\), with the usual zero-probability convention. Equation 4 may be negative on an individual transition. Averaging it under the actual joint distribution gives \(I(M_t; U_t \mid H_t)\); Equation 5 integrates the two binary variables at a given history and lies in [0, 1] bits. No fitted likelihood model is required here because the simulator exposes the policies.

For A, sent flow is \(A_t \to B_{t+1}\) and received flow is \(B_t \to A_{t+1}\). Their sum is a bidirectional total, not a new joint mutual information. A's sent score equals B's received score by definition. We call the measure conditional predictive information: evaluating branches does not make the naturally weighted score a universal causal-effect measure. In particular, a publicly predictable sender has \(w \in \{0, 1\}\) and zero natural information even when the receiver would react differently to an off-policy move.

3Results I: cooperation distribution of fixed strategies

We used Axelrod-Python 4.14.0 reconstructions of the fourteen first-tournament submissions plus Random [Knight et al., 2016, Axelrod-Python developers, 2026]. Every unordered pairing, including self-play, received 100 independent 200-round repetitions: 120 pairings, 12,000 matches, and 2.4 million rounds. There was no added action noise; the horizon was known. Cooperation across the strategy set weights each of fifteen opponents equally, averaging both seats for self-play.

The High group contains Nydegger, Grofman, Tit for Tat, Shubik, Stein–Rapoport, and Tideman–Chieruzzi; the Middle group contains Davis, Friedman/Grudger, Graaskamp, Random, and Anonymous; the Low group contains Downing, Tullock, Feld, and Joss. Nydegger cooperates in 99.6% of its actions under this opponent mixture, while Joss cooperates in 28.7%. These rankings do not order payoff: Stein–Rapoport has the highest mean payoff in this simulation, approximately 2.606 per round. The three bins remain fixed in all subsequent profile analyses.

This distribution establishes the empirical reference, not a universal taxonomy. For example, Davis is Middle overall but cooperates throughout against several other strategies. The original source code is not available in the library; its reconstructions retain documented interpretive choices and endgame rules. Our Anonymous implementation behaves marginally like fair random choice, so the strategy set contains two similar stochastic baselines.

4Results II: the information-flow spectrum

We scored ten recorded games per unordered pairing at all 199 transitions in both directions: 1,200 games and 477,600 directional observations. The complete pass reproduces all 48,000 previously sampled scores at their original checkpoints. Within a group cell, pairings receive equal weight. Same-group cells pool both seats; reversing an off-diagonal cell relabels the same games. The nine ordered cells therefore correspond to six disjoint sets of games.

High–High has zero measured flow throughout and 99.48% mutual cooperation. Mixed groups are directionally asymmetric: Middle sends 0.150251 bits to High while High sends 0.000372 bits to Middle. Middle sends 0.137473 bits to Low, whereas Low sends 0.019691 bits to Middle. A highly cooperative strategy can thus receive substantial predictive information from a less cooperative partner while sending little in return.

The informative distinction is a profile, not "more flow means more cooperation." Middle sends the most information under the equal-group mixture; High receives the most. Middle–Middle and Low–Low have positive full-game means but nearly zero late-game totals: approximately 0.000107 and 0.000169 bits over rounds 101–199. Thus the distinctive full-horizon High–High pattern cannot be generalized into a claim that every near-zero-flow interaction is cooperative. These summaries motivate a training experiment rather than establish its outcome.

5Results III: training with intrinsic flow, entropy, and prediction

5.1 Objectives and architecture

The original objective pools both directional flows before applying its penalties. If \(s\) and \(r\) are the encounter's mean sent and received flow, pooling both player perspectives gives \(\bar\iota = (s + r)/2\). The implemented loss is

The pooled balance term is identically zero; it does not independently force A's outgoing and incoming flow to agree. Keeping A's directions separate would give \(s^2 + r^2 + (s - r)^2\), a different loss that exceeds Equation 6 by \(\tfrac{3}{2}(s - r)^2\).

We add action entropy \(H\) and opponent-prediction improvement \(G\):

Here \(z_{i,t}\) is START or the previous own/opponent action pair. The online predictor uses state-conditioned Beta–Bernoulli counts with a Beta(1, 1) prior; the baseline uses state-independent counts. Both forecasts precede the observed action, and counts update afterward. \(G\) is a prequential log-score advantage over a base-rate predictor, not necessarily an improvement from one training iteration to the next. Minimizing positive \(H\) favors decisive actions; minimizing \(-G\) favors actions predictable from the richer model.

Each learner has 30 parameters: for each of five states, a linear logit uses an intercept and five centered posterior cooperation forecasts, followed by a sigmoid with a \(10^{-6}\) probability floor. Although the explicit action state is memory-one, the accumulated beliefs carry longer history. We compare \(F + H - G\), \(F + H\), \(F - G\), \(H\), \(-G\), and \(F\), all with unit component weights. Fixed opponents have observational shadow predictors for encounter-level \(G\); those predictors never change the fixed strategy's actions.

5.2 One learner against a fixed strategy, then two learners

The ablation contains twenty seeds, six objectives, and four training partners: fixed Nydegger, Feld, Joss, or an independently parameterized learner B. Each condition runs 1,000 Adam updates at learning rate 0.005, using 32 games per update split across seat orientations. Architecture, initial weights, update budget, and predictor are matched across objectives. Policy gradients account for sampled-history dependence as well as the direct derivative of the information terms. Independent half-batch products estimate the squared population flow without adding a sampling-variance penalty. Evaluation freezes the actors and uses fresh games in both orientations.

Against fixed partners, \(F\) alone yields 66.14%, 77.75%, and 62.67% learner cooperation against Nydegger, Feld, and Joss, respectively. Mutual cooperation is 63.64%, 46.57%, and 37.80%. Relative to matched initial policies, the increases in learner cooperation are 17.17, 27.45, and 12.43 percentage points; seed-bootstrap 95% intervals are [8.71, 25.61], [16.56, 36.92], and [0.11, 24.56]. Adding \(H\) or \(G\) does not improve these means. The full loss reaches only 29.81%, 12.31%, and 13.36% mutual cooperation.

The two-learner result is different. Under \(F\) alone, A cooperates 50.03%, B 50.96%, and both cooperate 25.48%. Mean action entropy remains 0.9701 bits even though \(F\) falls to \(2.99 \times 10^{-7}\). The full loss reduces entropy to 0.0663 bits but achieves 22.87% mutual cooperation. Across the six controls, mutual cooperation ranges from 19.69% to 25.48%. Thus reducing information flow can remove conditional dependence without creating cooperation, and adding predictability or decisiveness does not resolve that problem in this ablation.

This is a substantive constraint on the hypothesis: the information objective is behaviorally consequential against particular fixed partners, but it does not identify cooperative outcomes by itself. It motivates retaining the reference interaction's directional and temporal structure rather than optimizing only toward zero.

6Results IV: temporal signatures and function families

6.1 Direct fits to the nine directed group profiles

We fit each unsmoothed 199-point observed group curve using six candidates: zero, constant, exponential approach, logistic transition, Gaussian pulse, and an asymmetric rise–decay pulse. Five-fold validation holds out whole games, stratified within each strategy pairing (eight training and two held-out games per pair). We select the fewest-parameter family within one standard error of the best mean validation RMSE, then refit on all games. Folds assess new repetitions of the same strategy pairs, not unseen strategies or future time intervals.

The profiles have visibly different shapes. High→High is zero; High→Low selects a constant. Middle→High and Middle→Low select exponential approaches. Middle→Middle selects a sharp logistic decline. Low→High selects an asymmetric rise–decay pulse; Low→Middle and Low→Low select Gaussian pulses. High→Middle selects a narrow Gaussian event.

These are empirical model classes rather than proven universal laws. The High→Middle Gaussian width and Middle→Middle logistic width hit the half-round lower bound. Pair-level decomposition traces the narrow event and abrupt decline to Graaskamp's programmed testing schedule. The Middle→Low exponential time constant exceeds the 199-transition window, limiting interpretation of its asymptote. A class can recur in several groups, so no one-to-one map from function family to cooperative intent has been established.

6.2 Fresh fits to the equal-opponent-mixture curves

For each A group we first average the observed curves over B groups with weights 1/3, then fit a new model directly to each aggregate. We independently fit sent, received, and their observed sum. We do not average previously fitted group components. Consequently, an independently fitted combined curve need not equal the sum of its two fitted directional curves, although the observed data obey that identity exactly.

For High, sent flow selects a constant and received flow an exponential. For Middle, sent flow selects an exponential while received flow selects a Gaussian. For Low, sent flow selects a rise–decay pulse and received flow an exponential. These fitted signatures give a compact description of distinct empirical interaction regimes. They motivate a profile-matching objective, but descriptive fit quality does not itself establish that training toward a shape will reproduce the behavior that generated it.

7Results V: using a fitted profile as a learning target

7.1 Pilot objectives and matched search

The pilot tests one concrete target: the High group's equal-opponent-mixture sent and received profiles. Their fitted forms, in bits per next action, are approximately

This reference is not zero in both directions: High agents receive information from a mixed field of opponents. It captures the reference's asymmetry rather than only the High–High cell.

Let \(s_b(\theta)\) and \(r_b(\theta)\) denote the learner's expected directional flow in block \(b\) under the training opponent mixture. Let \(\omega_b\) equal the block's number of transitions divided by 199, and bars denote these weighted means. The three minimized objectives are

Combined flow is not added as a third error term because it is determined by the two directions. This is a comparison of zero-seeking, mean-target, and curve-target objectives, not yet a tournament of every function family as a loss.

The learner has five sigmoid logits, one for START and each previous joint-action state. It has no opponent identity, belief vector, or time input. Five matched seeds use cross-entropy-method search: eight generations, twelve candidates per generation, four elites, and the same initialization and perturbation streams across objectives. Each candidate plays two independent batches of fifteen games, one game per fixed strategy in each batch. High/Middle/Low opponent groups each receive weight 1/3, uniformly within group. This yields 43,200 training games across all three objectives.

Training evaluates Equation 5 at one uniformly selected transition per block. Independent-batch products estimate squared profile discrepancy; this avoids rewarding low Monte Carlo variance instead of the intended mean profile. The final search mean, not a cooperation-selected checkpoint, is evaluated. Eight fresh 200-round games per opponent, seed, and objective include all fifteen training identities plus four unseen identities: Cooperator, Defector, Win–Stay–Lose–Shift, and Generous Tit for Tat. Including the untrained baseline, evaluation contains 3,040 games. There is no learned-versus-learned curve-loss condition.

7.2 Cooperation and attained information profiles

Curve matching increases mutual cooperation by 14.25 percentage points over \(F\) on the original strategy set and 20.11 points on unseen strategies. Exploratory paired seed-bootstrap 95% intervals are [0.85, 34.38] and [1.83, 42.29] points. Mean-only matching is higher still: curve-minus-mean differences are −3.70 points on the original strategy set (interval [−10.46, 0.85]) and −2.72 on unseen strategies ([−8.23, 0.73]). Five seeds do not support a strong claim of superiority or equivalence between those two objectives.

Against Nydegger, Feld, and Joss, curve matching achieves 77.48%, 28.29%, and 34.39% mutual cooperation, versus 46.89%, 18.93%, and 24.05% for the matched \(F\) pilot. Mean matching reaches 78.56%, 30.89%, and 36.74%. Both reference objectives improve these means. The earlier ablation's \(F\) values cannot serve as matched controls here: the policy class, optimizer, training opponent mixture, and seed count changed.

Curve matching reduces the plug-in profile error by about 89% relative to initialization. Yet mean outgoing flow remains 0.07544 versus a 0.00679 target, and incoming flow remains 0.03873 versus approximately 0.0737. Seed variation is large: curve-trained A's cooperation against the original strategy set ranges from 14.7% to 88.3%. Moreover, its mean own cooperation falls from 52.9% initially to 49.1% while mutual cooperation rises. The result concerns improved joint outcomes, not a uniform increase in each agent's willingness to choose C.

8Related work and interpretation

The informational theory of psychological flow. Melnikoff et al. [2022] propose that subjective flow increases with mutual information between means and desired end states, supported across five experiments. Their question concerns immersion in a task. Our question concerns predictive dependence between agents' actions. The connection is methodological: both use an information-theoretic relation rather than an externally assigned payoff to describe organized behavior. However, reducing our inter-agent conditional information is not a deduction from their theory, which emphasizes stronger means–ends association. We neither measure subjective experience nor claim that these programs experience psychological flow.

Information transfer and influence. Our time-directed conditional score is related to transfer entropy [Schreiber, 2000], but its conditioning set additionally includes the known policies and receiver's current action. Jaques et al. [2019] use counterfactual effects on other agents as an intrinsic social-influence reward and connect that reward to mutual information. Their approach seeks influence; our initial loss seeks low residual predictive information, and our later loss seeks a measured directional profile. These are different objective choices within a related framework, not evidence that one sign is universally correct. Empowerment maximizes action-to-state channel capacity [Klyubin et al., 2005]; it differs both in variables and in maximizing over an action distribution. Our score uses the sender's actual policy distribution.

Modeling a learning partner. Opponent-learning awareness explicitly anticipates other learners' parameter updates [Foerster et al., 2018]. Our Bayesian predictor only forecasts observed actions and does not differentiate through the partner's future learning rule. Its improvement term is an intrinsic predictability signal, not an implementation of opponent-learning awareness. This distinction matters because the fixed-partner success does not imply success when both policies adapt.

9What the evidence supports, and what remains open

The results support three conclusions. First, under the declared strategy set and weighting, cooperation regimes have distinguishable directional and temporal profiles. Second, information-based losses can change cooperation without expert action targets or payoff rewards. Third, using a high-cooperation reference profile improves mutual cooperation over the original zero-seeking loss in the matched pilot, including four held-out strategy identities. The temporal target has not yet outperformed its mean-only counterpart.

Two different training claims must remain separate. The F/H/G ablation is self-supervised in the operational sense that its signals come from the agents' own simulated probabilities and observed actions, without cooperative-action labels or a selected behavior distribution. The curve pilot is reference-guided: choosing a High-group target uses earlier cooperation measurements and explicitly specifies a distributional statistic of interaction. It avoids action imitation, but cannot support the statement that no reference distribution is supplied. Neither experiment establishes predictable behavior of a many-agent swarm.

A decisive next test would hold architecture, optimizer, opponent coverage, and compute fixed while comparing F/H/G controls, mean targets, and multiple temporal targets in both learned–fixed and learned–learned settings. Profile selection should use training-only data; evaluation should include held-out strategies, changed horizons, and perturbation recovery. Persistent cooperation, predictive calibration, and stability across seeds should be reported separately. Otherwise a predictable but uncooperative equilibrium can satisfy an intrinsic objective.

Conclusion. Information flow is a promising main metric for learning organized interaction without supervising individual action choices. Our current evidence demonstrates measurable profile structure and action-label-free improvements in cooperation against fixed partners. It motivates the broader hypothesis that collaboration and predictable swarm behavior can be trained through self-supervised information objectives; reliably achieving that outcome between adaptive agents, without a selected reference profile, remains the research objective rather than an established result.