The CVPR 2019 Conceptual Captions Challenge is based on two separate test sets:
T1) a blind test set that participants do not have direct access to. This test set consists of about 12.5K image & captions pairs approved by human annotators, but otherwise preserving the distribution and style of images present in the Conceptual Captions dataset. You must submit your model to get your official scores. Submissions are handled via the Google Container Registry. Your model must be packaged in a Docker image and then uploaded to the Google Container Registry so that we can run it against the test set. Please see Prerequisites below for detailed info.
The results obtained by the submitted model(s) are made available on a leaderboard as part of this competition. You will have access to the global leaderboard that will rank the successful submissions we receive.
T2) an open test set that participants have access to and can download. This test set consists of 1000 images, taken from the Open Image Dataset. You must run the same model (the one submitted for T1) on the T2 images, and submit the resulting captions to us.
If your model submitted for T1 test set scores within the top-10 on the Challenge leaderboard and you have provided us the captions for your T2 open test set, we will evaluate your T2 captions using human evaluators for accuracy. The final model rankings for the Challenge will be based on the rankings achieved based on the human evaluation results of the T2 captions.
Submitting a Model
The submission platform accepts two official submission for each participant (unlimited test submission can be attempted to make sure the submission mechanics work as intended). You can also use the provided T2 images to make sure that you get the right outputs from your docker image.
Each submission must process all T1 test images in under 24 hours or they will be considered failed. The execution environment has 32GB available memory and limited disk. Exceeding the amount of storage on the system will be considered a failed submission.
Sign up for a Google Cloud Platform account at cloud.google.com. Few important points below
If this is a new account, you will be following the sign up process which may require billing information.
For the purpose of this competition, your GCP account will be used only for the storage of the container image(s), which allows you to maintain all ownership of your container image(s). The compute cost of model evaluation will not be charged to your GCP account. You can start with free credit, if not already exhausted.
Post-model evaluation, canceling your Google cloud billing will not eliminate your leaderboard entries or scores.
Install Cloud SDK. This is needed to interact with Google Cloud Project using command-line tools such as gcloud.
Install Docker. This is needed to build a Docker image for your model.
Building your Docker image
You must submit your model as a Docker image. You are allowed to use whatever software dependencies you want, but those dependencies must be included in your Docker image. Let's say your model is a Tensorflow model. For this, you can use the official tensorflow Docker container as the base container image:
The first line of this Dockerfile says to use the official Tensorflow Docker image as the starting point of the image. The second line says to add the contents of a directory called "conceptual-captions" to a folder called "/conceptual-captions" inside the image. Read the Docker manual for more details on how to use Dockerfiles.
The folder "/conceptual-captions" is expected to contain a script called "submission.sh". The submission.sh script should call your model to get captions for each input image, and write them to file. Here is an example submission.sh.
If you have configured your Docker image correctly, you should be able to run docker build to create your Docker image. Example:
To help you get started, we have created a skeleton you can use as a starting point. See the official GitHub page for more details.
Testing your Docker image
Once you have built your Docker image, you can test it against a small chunk of test data using the following command:
Where <DOCKER_IMAGE_ID> is the hex identifier of your image, as printed by docker build (60f7545b8562 in the example above).
You should see exactly one caption printed to /dev/stdout per image in test.images.tsv.
Pushing your Docker image
Once you are able to build your Docker image, you must push it to Google Container Registry.
Create your Google Cloud Platform project if you haven't already by visiting the console.
Obtain the numeric PROJECT_ID for your Google Cloud Platform project. This should be visible from your project's homepage.
Build and push to Google Container Registry:
Take note of the value of the --tag argument. You will use that when you create your submission.
Grant Access to our Service Account
One last step is required. You must grant our service account read access to your project's Docker images. Otherwise your Docker images will be private and we won't be able to run them.
Go to the storage tab in the gcloud console.
Locate the artifacts bucket. It should have a name like artifacts.<project-name >.appspot.com
Click the dropdown for the bucket and select "Edit Bucket Permissions".
Grant Storage Object Viewer permissions to the following user: firstname.lastname@example.org