Challenge

The CVPR 2019 Conceptual Captions Challenge is based on two separate test sets:

T1) a blind test set that participants do not have direct access to. This test set consists of about 12.5K image & captions pairs approved by human annotators, but otherwise preserving the distribution and style of images present in the Conceptual Captions dataset. You must submit your model to get your official scores. Submissions are handled via the Google Container Registry. Your model must be packaged in a Docker image and then uploaded to the Google Container Registry so that we can run it against the test set. Please see Prerequisites below for detailed info.

The results obtained by the submitted model(s) are made available on a leaderboard as part of this competition. You will have access to the global leaderboard that will rank the successful submissions we receive.

T2) an open test set that participants have access to and can download. This test set consists of 1000 images, taken from the Open Image Dataset. You must run the same model (the one submitted for T1) on the T2 images, and submit the resulting captions to us.

If your model submitted for T1 test set scores within the top-10 on the Challenge leaderboard and you have provided us the captions for your T2 open test set, we will evaluate your T2 captions using human evaluators for accuracy. The final model rankings for the Challenge will be based on the rankings achieved based on the human evaluation results of the T2 captions.

Submitting a Model

The submission platform accepts two official submission for each participant (unlimited test submission can be attempted to make sure the submission mechanics work as intended). You can also use the provided T2 images to make sure that you get the right outputs from your docker image.

Each submission must process all T1 test images in under 24 hours or they will be considered failed. The execution environment has 32GB available memory and limited disk. Exceeding the amount of storage on the system will be considered a failed submission.

Please visit the Workshop's Submission Challenge page for more details and to make your submission (before April 26, 2019).

Prerequisites

    1. Sign up for a Google Cloud Platform account at cloud.google.com. Few important points below

      • If this is a new account, you will be following the sign up process which may require billing information.

      • For the purpose of this competition, your GCP account will be used only for the storage of the container image(s), which allows you to maintain all ownership of your container image(s). The compute cost of model evaluation will not be charged to your GCP account. You can start with free credit, if not already exhausted.

      • Post-model evaluation, canceling your Google cloud billing will not eliminate your leaderboard entries or scores.

  1. Install Cloud SDK. This is needed to interact with Google Cloud Project using command-line tools such as gcloud.

  2. Install Docker. This is needed to build a Docker image for your model.

Building your Docker image

You must submit your model as a Docker image. You are allowed to use whatever software dependencies you want, but those dependencies must be included in your Docker image. Let's say your model is a Tensorflow model. For this, you can use the official tensorflow Docker container as the base container image:

FROM tensorflow/tensorflow:latest ADD conceptual-captions /conceptual-captions/

The first line of this Dockerfile says to use the official Tensorflow Docker image as the starting point of the image. The second line says to add the contents of a directory called "conceptual-captions" to a folder called "/conceptual-captions" inside the image. Read the Docker manual for more details on how to use Dockerfiles.

The folder "/conceptual-captions" is expected to contain a script called "submission.sh". The submission.sh script should call your model to get captions for each input image, and write them to file. Here is an example submission.sh.

#!/bin/bash # # submission.sh: The script to be launched in the Docker image. # # Usage: submission.sh <image_basename_file> <image_dir> <output_file> # image_basename_file: Path to file containing image file basenames (e.g. "123.jpg"), one per line # image_dir: Path to directory containing image files (e.g. "/tmp/image_dir") # output_file: Path to file where image captions should be written, one per line # # Sample usage: # submission.sh images.txt image_dir captions.txt IMAGE_BASENAME_FILE=$1 IMAGE_DIR=$2 OUTPUT_FILE=$3 for image in $(cat $IMAGE_BASENAME_FILE); do # YOUR CODE HERE! # # For example: # python generate_caption.py --image_path=$IMAGE_DIR/$image >> $OUTPUT_FILE echo "dummy caption" >> $OUTPUT_FILE done

If you have configured your Docker image correctly, you should be able to run docker build to create your Docker image. Example:

$ docker build . Sending build context to Docker daemon 3.072kB Step 1/2 : FROM tensorflow/tensorflow:latest latest: Pulling from tensorflow/tensorflow 297061f60c36: Pull complete e9ccef17b516: Pull complete dbc33716854d: Pull complete 8fe36b178d25: Pull complete 686596545a94: Pull complete ed66f2c5f3d9: Pull complete 8405b6c3f141: Pull complete 070615ca3a03: Pull complete 306ac2321f8e: Pull complete c30111bc1e74: Pull complete 7aa552c3f7f7: Pull complete 4db41af3662a: Pull complete bf5fbadacf01: Pull complete Digest: sha256:1cc84937252fcc6e8521901cbbe180c7a93300792aceb0da8a53a5728360320a Status: Downloaded newer image for tensorflow/tensorflow:latest ---> e7a53807ee54 Step 2/2 : ADD src /conceptual-captions/ ---> 60f7545b8562 Successfully built 60f7545b8562

To help you get started, we have created a skeleton you can use as a starting point. See the official GitHub page for more details.

Testing your Docker image

Once you have built your Docker image, you can test it against a small chunk of test data using the following command:

docker run -a stdin -a stdout -a stderr <DOCKER_IMAGE_ID> \ bash /conceptual-captions/submission.sh \ /conceptual-captions/test_images.txt \ /conceptual-captions/test_images \ /dev/stdout

Where <DOCKER_IMAGE_ID> is the hex identifier of your image, as printed by docker build (60f7545b8562 in the example above).

You should see exactly one caption printed to /dev/stdout per image in test.images.tsv.

Pushing your Docker image

Once you are able to build your Docker image, you must push it to Google Container Registry.

    1. Create your Google Cloud Platform project if you haven't already by visiting the console.

    2. Obtain the numeric PROJECT_ID for your Google Cloud Platform project. This should be visible from your project's homepage.

    3. Build and push to Google Container Registry:

gcloud auth login gcloud config set project [PROJECT_ID] gcloud builds submit --tag gcr.io/[PROJECT_ID]/my-submission .

Take note of the value of the --tag argument. You will use that when you create your submission.

Note: Read the quickstart guide for more details on how to use Google Container Registry.

Grant Access to our Service Account

One last step is required. You must grant our service account read access to your project's Docker images. Otherwise your Docker images will be private and we won't be able to run them.

  1. Go to the storage tab in the gcloud console.

  2. Locate the artifacts bucket. It should have a name like artifacts.<project-name >.appspot.com

  3. Click the dropdown for the bucket and select "Edit Bucket Permissions".

  4. Grant Storage Object Viewer permissions to the following user: mljam-compute@mljam-205019.iam.gserviceaccount.com