vSwarm's benchmarks cannot run on the firecracker-container

[smallpip]

HI,I opened firecracker and want to achieve Figure in ASPLOS '21 using vSwarm's benchmarks, but it couldn't be performed. The reason may be that I am not modifying the yaml file correctly.

So I wanna know how to modified vSwarm yaml or find correct example yaml for firecracker. Mainly I will not set parameters for nested virtualization

Below is my according toconfigs/knative_workloads/helloworld.yamlmodified fibonacci.yaml .

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: fibonacci-python
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: docker.io/vhiveease/relay:latest
          ports:
            - name: h2c
              containerPort: 50001
          env:
            - name: GUEST_PORT # Port on which the firecracker-containerd container is accepting requests
              value: "50051"
            - name: GUEST_IMAGE # Container image to use for firecracker-containerd container
              value: "docker.io/vhiveease/fibonacci-python:latest"
            - name: VALUE
              value: "10"
            - name: GENERATOR
              value: "linear"
            - name: UPPERBOUND
              value: "10"
            - name: LOWERBOUND
              value: "1"

and then this is the result of the invoke

  DEBU[2023-08-10T07:40:22.853805056Z] Debug logging is enabled                     
  INFO[2023-08-10T07:40:22.853891548Z] Reading the endpoints from the file: endpoints.json 
  DEBU[2023-08-10T07:40:22.854011818Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
  DEBU[2023-08-10T07:40:23.854346681Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
  DEBU[2023-08-10T07:40:24.854455445Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
  DEBU[2023-08-10T07:40:25.854585940Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
  DEBU[2023-08-10T07:40:26.854758531Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
  INFO[2023-08-10T07:40:27.854957023Z] Issued / completed requests: 5, 0            
  INFO[2023-08-10T07:40:27.854999480Z] Real / target RPS: 0.00 / 1                  
  INFO[2023-08-10T07:40:27.855012060Z] Experiment finished!                         
  INFO[2023-08-10T07:40:27.855022200Z] The measured latencies are saved in rps0.00_lat.csv 

[leokondrashov]

Hi, please provide additional information about your setup: what commands were used to setup the vHive, output of kubectl get pods -A, any errors that happened before the attempts to deploy the vSwarm functions. Please check if default vHive .yaml files are operational before moving to vSwarm as well, since they were definitely made for firecracker.

Additionally, we recently noticed that our firecracker setup is not working properly (#781). Your problem with deployment might be caused by that as well.

Directly answering your question, there is an issue in vSwarm repo right about that (vhive-serverless/vSwarm#257). Please try that if no other problems in the configuration are present.

[smallpip]

thanks,I think there is no problem in my deployment, because kubectl get pods are running normally after the firecracker is started, and there is no problem in executing the sample helloworld.yaml.

this is a part of firecracker.stderr

time="2023-08-10T02:45:06.667092805Z" level=debug msg="using namespace: firecracker-containerd"
time="2023-08-10T02:45:06.667394597Z" level=debug msg="starting containerd-shim-aws-firecracker" vmID=2
time="2023-08-10T02:45:06.708086351Z" level=info msg="starting signal loop" namespace=firecracker-containerd path=/var/lib/firecracker-containerd/shim-base/firecracker-containerd/2 pid=92580
time="2023-08-10T02:45:06.708927839Z" level=info msg="creating new VM" runtime=aws.firecracker vmID=2
time="2023-08-10T02:45:06.709393238Z" level=info msg="Called startVMM(), setting up a VMM on firecracker.sock" runtime=aws.firecracker
time="2023-08-10T02:45:06.723143058Z" level=info msg="refreshMachineConfiguration: [GET /machine-config][200] getMachineConfigurationOK  &{CPUTemplate:T2 HtEnabled:0xc000707a43 MemSizeMib:0xc000707a38 TrackDirtyPages:false VcpuCount:0xc000707a30}" runtime=aws.firecracker
time="2023-08-10T02:45:06.723406794Z" level=info msg="PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent " runtime=aws.firecracker
time="2023-08-10T02:45:06.723429711Z" level=info msg="Attaching drive /var/lib/firecracker-containerd/runtime/default-rootfs.img, slot root_drive, root true." runtime=aws.firecracker
time="2023-08-10T02:45:06.723851620Z" level=info msg="Attached drive /var/lib/firecracker-containerd/runtime/default-rootfs.img: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent " runtime=aws.firecracker
time="2023-08-10T02:45:06.723874877Z" level=info msg="Attaching drive /var/lib/firecracker-containerd/shim-base/firecracker-containerd/2/ctrstub0, slot MN2HE43UOVRDA, root false." runtime=aws.firecracker
time="2023-08-10T02:45:06.724078370Z" level=info msg="Attached drive /var/lib/firecracker-containerd/shim-base/firecracker-containerd/2/ctrstub0: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent " runtime=aws.firecracker
time="2023-08-10T02:45:06.724099873Z" level=info msg="Attaching NIC 2_tap (hwaddr 02:FC:00:00:00:01) at index 1" runtime=aws.firecracker
time="2023-08-10T02:45:06.750438920Z" level=info msg="startInstance successful: [PUT /actions][204] createSyncActionNoContent " runtime=aws.firecracker
time="2023-08-10T02:45:06.750470752Z" level=info msg="calling agent" runtime=aws.firecracker vmID=2
time="2023-08-10T02:45:07.550911549Z" level=info msg="successfully started the VM" runtime=aws.firecracker vmID=2
time="2023-08-10T02:45:07.551567233Z" level=debug msg="event forwarded" ns=firecracker-containerd topic=/firecracker-vm/start type=VMStart
time="2023-08-10T02:45:07.558189391Z" level=debug msg="prepare snapshot" key=2 parent="sha256:e172bf8795d813920c658a6772bb108238dc2cf13f1fc1ee1ac5c595c37da14a"
time="2023-08-10T02:45:07.560163043Z" level=debug msg=prepare key=firecracker-containerd/24/2 parent="firecracker-containerd/23/sha256:e172bf8795d813920c658a6772bb108238dc2cf13f1fc1ee1ac5c595c37da14a"
time="2023-08-10T02:45:07.560274697Z" level=debug msg="creating snapshot device 'fc-dev-thinpool-snap-13' from 'fc-dev-thinpool-snap-12'"
time="2023-08-10T02:45:07.683974704Z" level=debug msg="event published" ns=firecracker-containerd topic=/snapshot/prepare type=containerd.events.SnapshotPrepare
time="2023-08-10T02:45:07.686950594Z" level=debug msg="get snapshot mounts" key=2
time="2023-08-10T02:45:07.686987720Z" level=debug msg=mounts key=firecracker-containerd/24/2
time="2023-08-10T02:45:07.720838343Z" level=debug msg="event published" ns=firecracker-containerd topic=/containers/create type=containerd.events.ContainerCreate
time="2023-08-10T02:45:07.724268292Z" level=debug msg="get snapshot mounts" key=2
time="2023-08-10T02:45:07.724332382Z" level=debug msg=mounts key=firecracker-containerd/24/2
time="2023-08-10T02:45:07.761568159Z" level=debug msg=StartShim runtime=aws.firecracker task_id=2
time="2023-08-10T02:45:07.762730490Z" level=debug msg="create VM request: VMID:\"2\" "
time="2023-08-10T02:45:07.762769021Z" level=debug msg="using namespace: firecracker-containerd"
time="2023-08-10T02:45:07.763336151Z" level=info msg="successfully started shim (git commit: 4a8adae2660e3db2ea9b5b7b057b4388b9f5bab8)." runtime=aws.firecracker task_id=2 vmID=2
time="2023-08-10T02:45:07.765905871Z" level=info msg="PatchGuestDrive successful" runtime=aws.firecracker
time="2023-08-10T02:45:07.813486851Z" level=debug msg="garbage collected" d=2.131395ms
time="2023-08-10T02:45:07.892138197Z" level=info msg="successfully created task" ExecID= TaskID=2 pid_in_vm=718 runtime=aws.firecracker vmID=2
time="2023-08-10T02:45:07.892439234Z" level=debug msg="event forwarded" ns=firecracker-containerd topic=/tasks/create type=containerd.events.TaskCreate
time="2023-08-10T02:45:07.898258636Z" level=debug msg="event forwarded" ns=firecracker-containerd topic=/tasks/start type=containerd.events.TaskStart

My question is the same as vhive-serverless/vSwarm#257, I want to know how to pass args variables to env, I see vhive-serverless/vSwarm#257 doesn't give an clear answer.

As for #781, I by changing configs/calico/canal.yaml has increased the following commands, solved the problem

sed -i '4548i\            - name: IP_AUTODETECTION_METHOD\n              value: "skip-interface=br.*"'

[leokondrashov]

Can you please create a PR for the #781, since you got it fixed? We'll appreciate the contribution.

Have you tried doing it that way (taken almost exactly from vhive-serverless/vSwarm#257)? If so, what exactly is happening to the container (failed to create, created but unresponsive or respond with some error)? Please provide the output of kubectl describe pod <name of created pod> and kubectl get pods.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: aes-python
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: crccheck/hello-world:latest # Stub image. See https://github.com/ease-lab/vhive/issues/68
          ports:
            - name: h2c # For GRPC support
              containerPort: 50051
          env:
            - name: GUEST_PORT # Port on which the firecracker-containerd container is accepting requests
              value: "50051"
            - name: GUEST_IMAGE # Container image to use for firecracker-containerd container
              value: "docker.io/vhiveease/fibonacci-python:latest"

[smallpip]

based on your yaml file,this is the execution result,same as before.

ubuntu@ip-172-31-29-88:~$ kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
fibonacci-python-00001-deployment-554d595db8-66htn   2/2     Running   0          6m37s
fibonacci-python-00001-deployment-554d595db8-x5kdn   2/2     Running   0          6m36s

ubuntu@ip-172-31-29-88:~$ kubectl describe pod fibonacci-python-00001-deployment-554d595db8-66htn
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m41s  default-scheduler  Successfully assigned default/fibonacci-python-00001-deployment-554d595db8-66htn to ip-172-31-28-224
  Normal  Pulling    7m40s  kubelet            Pulling image "index.docker.io/crccheck/hello-world@sha256:0404ca69b522f8629d7d4e9034a7afe0300b713354e8bf12ec9657581cf59400"
  Normal  Pulled     7m39s  kubelet            Successfully pulled image "index.docker.io/crccheck/hello-world@sha256:0404ca69b522f8629d7d4e9034a7afe0300b713354e8bf12ec9657581cf59400" in 778.722193ms
  Normal  Created    7m24s  kubelet            Created container user-container
  Normal  Started    7m24s  kubelet            Started container user-container
  Normal  Pulling    7m24s  kubelet            Pulling image "ghcr.io/vhive-serverless/queue-39be6f1d08a095bd076a71d288d295b6@sha256:41259c52c99af616fae4e7a44e40c0e90eb8f5593378a4f3de5dbf35ab1df49c"
  Normal  Pulled     7m23s  kubelet            Successfully pulled image "ghcr.io/vhive-serverless/queue-39be6f1d08a095bd076a71d288d295b6@sha256:41259c52c99af616fae4e7a44e40c0e90eb8f5593378a4f3de5dbf35ab1df49c" in 1.047158299s
  Normal  Created    7m23s  kubelet            Created container queue-proxy
  Normal  Started    7m23s  kubelet            Started container queue-proxy
  
ubuntu@ip-172-31-29-88:~$ kubectl describe pod fibonacci-python-00001-deployment-554d595db8-x5kdn
  Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned default/fibonacci-python-00001-deployment-554d595db8-x5kdn to ip-172-31-28-224
  Normal  Pulling    13m   kubelet            Pulling image "index.docker.io/crccheck/hello-world@sha256:0404ca69b522f8629d7d4e9034a7afe0300b713354e8bf12ec9657581cf59400"
  Normal  Pulled     13m   kubelet            Successfully pulled image "index.docker.io/crccheck/hello-world@sha256:0404ca69b522f8629d7d4e9034a7afe0300b713354e8bf12ec9657581cf59400" in 354.384392ms
  Normal  Created    12m   kubelet            Created container user-container
  Normal  Started    12m   kubelet            Started container user-container
  Normal  Pulling    12m   kubelet            Pulling image "ghcr.io/vhive-serverless/queue-39be6f1d08a095bd076a71d288d295b6@sha256:41259c52c99af616fae4e7a44e40c0e90eb8f5593378a4f3de5dbf35ab1df49c"
  Normal  Pulled     12m   kubelet            Successfully pulled image "ghcr.io/vhive-serverless/queue-39be6f1d08a095bd076a71d288d295b6@sha256:41259c52c99af616fae4e7a44e40c0e90eb8f5593378a4f3de5dbf35ab1df49c" in 1.097744043s
  Normal  Created    12m   kubelet            Created container queue-proxy
  Normal  Started    12m   kubelet            Started container queue-proxy

But the results of inovke are still incorrect

ubuntu@ip-172-31-29-88:~$ ./vSwarm/tools/invoker/invoker -port 50051 -time 5 -rps 1 -dbg
DEBU[2023-08-11T14:00:49.649717965Z] Debug logging is enabled                     
INFO[2023-08-11T14:00:49.649811536Z] Reading the endpoints from the file: endpoints.json 
DEBU[2023-08-11T14:00:49.649941401Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
DEBU[2023-08-11T14:00:50.650173364Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
DEBU[2023-08-11T14:00:51.650373983Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
DEBU[2023-08-11T14:00:52.650329385Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
DEBU[2023-08-11T14:00:53.650379630Z] Invoking: fibonacci-python.default.192.168.1.240.sslip.io:50051 
INFO[2023-08-11T14:00:54.650525660Z] Issued / completed requests: 5, 0            
INFO[2023-08-11T14:00:54.650569174Z] Real / target RPS: 0.00 / 1                  
INFO[2023-08-11T14:00:54.650581974Z] Experiment finished!                         
INFO[2023-08-11T14:00:54.650593670Z] The measured latencies are saved in rps0.00_lat.csv 

In addition,I don't understand. Does this env require no other values ？how to pass them 😅

[cvetkovic]

Could it be due to issue #68?

[ustiugov]

@cvetkovic I recall that Diego added support for the environment variables in #633 but left no docs on how to pass them. Do you have any working example?

[cvetkovic]

Environmental variables should be provided in the YAML file, on function deployment.

[ustiugov]

@cvetkovic do you have a sample YAML?

[cvetkovic]

AFAIK, Diego had some unit tests for this feature, but I don't see them committed.

You should just add additional variables as ITERATIONS_MULTIPLIER was added. That one works for sure and it's from the loader repo.

[ustiugov]

thanks a lot, that's helpful