profile
viewpoint

Ask questionsFlake: `iptables: create-instance-chains: iptables: Memory allocation problem.`

Flake Report

Occasionally it seems like Garden fails here when Concourse is trying to create a task container.

Examples

  • https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/testflight/builds/382
  • PCI team in the Toronto office saw this in their Concourse ( 4.2.1 ) environment as well

Observations

  • The worker metrics don't show any abnormalities in terms of memory utilization https://snapshot.raintank.io/dashboard/snapshot/2LRam0LlWhwwEMP38tr6MLNZ4FV6gaAL?orgId=2

Additional Context

@kcmannem has a stack dump from PCI's issue

Version Info

  • Concourse version: 4.2.1 & 5.0.0
  • Deployment type (BOSH/Docker/binary): BOSH
  • Infrastructure/IaaS: GCP
  • Did this used to work? 🤷‍♂️
concourse/concourse

Answer questions danail-branekov

Hi Concourse friends,

While investigating probably unrelated thing in Garden we managed to reproduce consistently that issue and maybe you would be interested in the scenario.

We created an AWS concourse deployment (if version matters, we used v.4.2.2) with one worker on a t3.nano VM (0.5G of memory) - we really wanted to put memory pressure on that worker. Then we created the following dummy task:

---
image_resource:
  type: docker-image
  source:
    repository: cfgarden/garden-ci
    tag: '2.0.11'

platform: linux

run:
  path: /bin/echo

Then we ran 8 parallel infinite loops running the same task over and over again.

for i in `seq 0 200` ; do echo Attempt $i && fly --target slab-ci e -p -c "<the task yml>"; done

Note that our endless loop runs fine when run in 4 instances

useful!
source:https://uonfu.com/
Github User Rank List