Started on automatic runs the P2P team can trigger themselves, and agreed the scope of the MVP.
Built a new 10ksim experiment that runs groups of nodes from any Docker image, reports which nodes failed and why, and can publish messages like the regression. Tested it on the cluster, including a 50 node run with a bootstrap node.
Tested a manual github workflow on a local branch that runs a config kept in another repository and shows the report on the run page. PR not created yet.
ift-ts:infra:
develp
Fixed the Develp.co website layout and merged the website revamp.
Added missing attestation alerts for mainnet fleets.
Fixed the missing ClickHouse plugin for some Grafana dashboards.
Fixed storage issues on GitHub runners.
lido
Deployed additional Lido CMv2 validator keys to master hosts.
Investigated a Web3Signer key reload failure caused by a bug.
Deployed the Lido CSM 0x02 Hoodi testnet operator and keys.
rocket
Completed migration of the oDAO smartnode from AWS to an OVH NixOS setup.
Created new dashboards, alerting, CI pipelines and tests.
nimbus
Rolled out upgrades and monitoring for the Sepolia Glamsterdam fork.
Improved source checkout for the nimbus-build-system dependency.
Fixed Nimbus builds on macOS broken by debug symbols.
Re-bootstrapped and deployed Nimbus nodes on a Mac M2 host.
Adjusted retries for ERA file generation on the fleet.
Enabled dumping invalid blocks from beacon nodes.
Completed ERE file generation for all fleets.
Deployed a Light Client on a Sepolia host for nimbus-eth1 sync.
Investigated Nimbus REST API test failures on Jenkins CI.
logos
Rolled out multiple Logos modules to logos.test and logos.dev.
Debugged a DHT issue and node logs being flooded with errors.
Fixed the fleet dashboard to show module versions correctly.
Continued organization validation for DigiCert Windows signing.
Prepared the LEZ 0.3.1 release deployment.
status
Resolved Android workspace cache collisions.
Lowered log verbosity for RPC tests in Jenkins CI.
Debugged go-sqlcipher issues in ARM builds.
sites
Debugged a status.app website crash caused by a Next.js configuration issue.
Improved site health checks to catch errors that still return HTTP 200.
Fixed CSP issues preventing podcast resources from loading.
Researched Apple IPA size limit warnings.
other notes
Decommissioned the misc.lez-legacy host.
ci
Worked on requiring signed commits across both GitHub Enterprises.
Rolled out Determinate Systems Nix to Jenkins and GitHub workers.
Upgraded Jenkins Master and removed deprecated Blue Ocean plugins.
Updated CI pipelines to use Graph Pipeline View for builds.
Deployed an Attic Nix cache at cache.nix.free.technology.
Tested Attic Nix cache uploads in Nimbus repositories.
hq
Completed the Vault upgrade to 1.21.4 on all cluster hosts.
Re-issued the status.im certificate for internal services.
Fixed Wazuh agent certificate deployment issues.
Started splitting Consul access by fleet using policies.
Continued Ubuntu upgrades across all fleets.
bi
Fixed Airbyte Twitter extraction and Airflow DAG notifications.
Added an OAuth proxy for Airbyte in Kubernetes to improve security.
Rotated all Airbyte tokens as a precaution against a potential breach.
Continued Engagement Bot features and Status SDK improvements.
logos-messaging/logos-delivery#4438 bug: the archive counts a message it already holds as written on Postgres and as an insert failure on sqlite
logos-messaging/logos-delivery#4437 question: GET /store/v3/messages reads a startTime or endTime of 0 or less as absent, and a negative pageSize as the maximum
logos-messaging/logos-delivery#4436 question: any —rate-limit entry replaces the default rate limits, so filter falls to 30/1m and unnamed protocols run without a limit
logos-messaging/logos-delivery#4435 question: filter UNSUBSCRIBE of a criterion the requester does not hold answers 200 or 404 depending on other peers, and its 404 can follow a partial removal
logos-messaging/logos-delivery#4434 bug: /filter/v2/subscriptions answers 200 “UNKNOWN” without pubsubTopic under static sharding, and its message cache ignores the result
logos-messaging/logos-delivery#4431 bug: /lightpush/v3/message answers 500 for INVALID_MESSAGE (420), and 413 or 500 for the same oversize message
logos-messaging/logos-delivery#4433 question: lightpush v3 client turns the service’s BAD_REQUEST for an undecodable request into “response failure, requestId mismatch”
logos-messaging/logos-delivery#4457 bug: POST /relay/v1/subscriptions joins a shard of another cluster or past —num-shards-in-network, and the node’s metadata then misreports its shards
logos-messaging/logos-delivery#4456 question: DELETE /relay/v1/auto/subscriptions of one content topic unsubscribes its whole shard, silencing the other content topics on it
logos-messaging/logos-delivery#4455 question: under autosharding —shard does not limit which shards the relay joins, yet the REST cache and —protected-shard still follow it
logos-messaging/logos-delivery#4454 question: —cluster-id=2 applies the logos.dev preset, whose nodes are on cluster 3, and replaces an explicit —preset=logos.test
logos-messaging/logos-delivery#4453 bug: a duplicate a node publishes goes through its own relay handlers again, and is answered as NoPeersToPublish
logos-messaging/logos-delivery#4485 bug: with a raised —max-msg-size, store sync never transfers a message above 150 KiB, and a few such messages stop the node’s transfers to every peer
logos-messaging/logos-delivery#4484 bug: a node that gets SIGTERM while store sync transfers messages to a peer never finishes stopping and is killed
logos-messaging/logos-delivery#4486 bug: a message encoded within —max-msg-size is not relayed or stored when the encoded RPC message gossipsub would send exceeds the limit, and the publish routes answer 200 or “no peers”
logos-messaging/logos-delivery#4483 bug: store sync transfer writes the pubsub topic as field 1 and the message as field 2, the reverse of the WAKU-SYNC specification
logos-messaging/logos-delivery#4482 bug: —preset=status.prod runs the 150KiB default of —max-msg-size instead of the 1024KiB of the preset, and warns at every start that the size was provided
logos-messaging/logos-delivery#4481 bug: a peer that connects in while the node’s own dial to it is in flight stays CannotConnect for as long as it is connected
logos-messaging/logos-delivery#4480 question: /admin/v1/peers lists a peer that dialled in over TCP at its connection’s source port, where nothing listens