2021 IEEE GRSS Data Fusion Contest: Track DSE
Detection of Settlements without Electricity
– End the Darkness –
The Challenge Track
The detection of settlements without electricity challenge track (Track DSE) of the 2021 IEEE GRSS Data Fusion Contest, organized by the Image Analysis and Data Fusion Technical Committee (IADF TC) of the IEEE Geoscience and Remote Sensing Society (GRSS), Hewlett Packard Enterprise, SolarAid, and Data Science Experts, aims to promote research in automatic detection of human settlements deprived of access to electricity using multimodal and multitemporal remote sensing data.
The task of Track DSE is to detect human settlements that do not have access to electricity using multimodal and multitemporal remote sensing data. Participants are required to submit binary classification maps and the performance will be evaluated using F1 score.
Competition Phases
Track DSE aims to promote innovation in detection of human settlements deprived of access to electricity, as well as to provide objective and fair comparisons among methods. The ranking is based on quantitative accuracy parameters computed with respect to undisclosed test samples. Participants will be given a limited time to submit their classification maps after the competition starts. The contest will consist of two phases:
- Phase 1: Participants are provided with training data (which includes reference data) and validation data (without reference data) to train and validate their algorithms. Participants can submit prediction results for the validation set to the Codalab competition website to get feedback on the performance. The performance of the last submission from each account will be displayed on the leaderboard. In parallel, participants submit a short description of the used approach to be eligible to enter Phase 2.
- Phase 2: Participants receive the test data set (without the corresponding reference data) and submit their binary classification maps within five days from the release of the test data set. After evaluation of the results, four winners per track are announced. Following this, they will have one month to write their manuscript that will be included in the IGARSS proceedings. Manuscripts are 4-page IEEE-style formatted. Each manuscript describes the addressed problem, the proposed method, and the experimental results.
Calendar
| December 3rd | Contest opening: release of training and validation data |
| January 4th | Evaluation server begins accepting submissions for validation data set |
| February 28th | Short description of the approach in 1-2 pages is sent to iadf_chairs@grss-ieee.org |
| March 1st | Release of test data; evaluation server begins accepting test submissions |
| March 5th | Evaluation server stops accepting submissions |
| March 12th | Source code is sent to iadf_chairs@grss-ieee.org for internal review. |
| March 26th | Winner announcement |
The Data
The contest dataset is composed of 98 tiles of 800×800 pixels, distributed respectively across the training, validation and test sets as follows: 60, 19, and 19 tiles. Each tile includes 98 channels from the below listed satellite images. Please note that all the images have been resampled to a Ground Sampling Distance (GSD) of 10 m. Thus each tile corresponds to a 64km2 area.
Satellite Data
Sentinel-1 polarimetric SAR dataset
2 channels corresponding to intensity values for VV and VH polarization at a 5×20 m spatial resolution scaled to a 10×10m spatial resolution.
- File name prefix: “S1A_IW_GRDH_*.tif”
- Size : 2.1 GB (float32)
- Number of images : 4
- Acquisition mode : Interferometric Wide Swath
- Native resolution : 5x20m
- User guide : link
Sentinel-2 multispectral dataset
12 channels of reflectance data covering VNIR and SWIR ranges at a GSD of 10 m, 20 m, and 60 m. The cirrus band 10 is omitted, as it does not contain ground information.
- File name prefix: “L2A_*.tif”
- Size : 6,2 GB (uint16)
- Number of images : 4
- Level of processing : 2A
- Native Resolution : link
- User guide : link
Landsat 8 multispectral dataset
11 channels of reflectance data covering VNIR, SWIR, and TIR ranges at a GSD of 30m and 100 m, and a Panchromatic band at a GSD of 15m.
- File name prefix: “LC08_L1TP_*.tif”
- Size : 8,5 GB (float32)
- Number of images : 3
- Sensor used : OLI and TIRS
- Native Resolution : link
- User guide : link
The Suomi Visible Infrared Imaging Radiometer Suite (VIIRS) night time dataset
The Day-Night Band (DNB) sensor of the VIIRS (Visible Infrared Imaging Radiometer Suite) provides on 1 channel, the global daily measurements of nocturnal visible and near-infrared (NIR) light at a GSD of 750 m. The VNP46A1 product is a corrected version of the original DNB data, and is at a 500m GSD resolution.
- File name prefix: “DNB_VNP46A1_*.tif”
- Size : 1,2 GB(uint16)
- Number of images : 9
- Product Name : VNP46A1’s 500x500m sensor radiance dataset
- Native resolution : 750m (raw resolution)
- User sheet : link
Semantic Labels
The provided training data is split across 60 folders named TileX, X being the tile number. Each folder includes 100 files. 98 files correspond to the satellite images listed earlier.
We also provide reference information (‘groundTruth.tif’ file) for each tile. Please note that the labeling has been performed as follows:
- Human settlement: If a building is present in a patch of 500×500m, this area is considered to have human settlement
- Presence of Electricity: If a fraction of a patch of 500×500m is illuminated, this area is considered to be illuminated regardless of the size of the illuminated area fraction.
The reference file (‘groundTruth.tif’) is 16×16 pixels large, with a resolution of 500m corresponding to the labelling strategy described above. The pixel values (1, 2, 3 and 4) correspond to the four following classes:
- 1: Human settlements without electricity (Region of Interest)
- 2: No human settlements without electricity
- 3: Human settlements with electricity
- 4: No human settlements with electricity
An additional reference file ( ‘groundTruthRGB.png’ ) is provided at 10m resolution in RGB for easier visualization.
| Class Number | Class Name | Color |
| 1 | Human settlements without electricity (Region of Interest) | ff0000 |
| 2 | No human settlements without electricity | 0000ff |
| 3 | Human settlements with electricity | ffff00 |
| 4 | No human settlements with electricity | b266ff |
Visual examples of the training data and the high-resolution classification map used in the contest are shown in Figure 1.
Figure 1: Visual examples of satellite images (a)-(c) and reference data (d) used in DFC2021 Track DSE.
We expect submissions to provide a binary classification of the Region of Interest, at a 500m GSD resolution (i.e. 16×16 pixels) as a GeoTIFF with the Byte (uint8) data type:
- 1: for human settlements without electricity (Region of Interest)
- 0: for any other class
The reference data for the validation and test sets remains undisclosed and will be used for the evaluation of the results.
Results, Awards, and Prizes:
- The first, second, third, and fourth-ranked teams in Track DSE will be declared as winners.
- The authors of the winning submissions will:
- Present their manuscripts in an invited session dedicated to the Contest at IGARSS 2021
- Publish their manuscripts in the Proceedings of IGARSS 2021
- Be awarded IEEE Certificates of Recognition
- The first, second, and third-ranked teams in Track DSE will receive $5,000, $3,000, and $2,000 (USD), respectively, as a special prize.
- The authors of the first and second-ranked teams in Track DSE will co-author a journal paper (in a limit of 3 co-authors per submission), which will summarize the outcome of the Contest and will be submitted with open access to IEEE JSTARS.
- Top-ranked teams will be awarded during IGARSS 2021, Brussels, Belgium in July 2021. The costs for open-access publication will be supported by the GRSS. The winner team prize is kindly sponsored by the organizing partners.
The rules of the game:
- All data can be requested by registering for the Contest at IEEE DataPort.
- To enter the contest, participants must read and accept the Contest Terms and Conditions.
- Participants of the contest are required to submit binary classification maps in raster format (similar to the tif files of the training set but with only two instead of four classes).
- For sake of visual comparability of the results, all classification maps shown in figures or illustrations should follow the color palette in the class table above.
- The classification results will be submitted to the CodaLab competition website for evaluation.
- Ranking between the participants will be based on the F1 score.
- Scoring will be performed on 16×16 tiles only. However, for the sake of visualization and for future applications estimates on the full resolution, i.e. 800×800, are also of great interest and can be submitted as supplemental material together with source code.
- The maximum number of trials of one team for each classification challenge is ten in the test phase.
- Submission server will be open from March 1, 2021. Deadline of classification result submission is March 5, 2021, 23:59 UTC – 12 hours (e.g., March 5, 2021, 7:59 in New York City, 13:59 in Paris, or 19:59 in Beijing).
- Each team needs to submit a short paper of 1–2 pages clarifying the used approach, the team members, their Codalab accounts, and one Codalab account to be used for the test phase by Feburary 28, 2021. Please send a paper to iadf_chairs@grss-ieee.org.
- While open sourcing the used software is strongly encouraged, each team needs to submit its source code by March 12, 2021 for internal review only. After the contest the winning solutions will have the opportunity to gain further visibility by making their solution available to promote a mapping project in Africa lead by Hewlett Packard Enterprise, SolarAid, and Data Science Experts.
- For the four winners, internal deadline for full paper submission is April 23, 2021, 23:59 UTC – 12 hours (e.g., April 23, 2021, 7:59 in New York City, 13:59 in Paris, or 19:59 in Beijing). IGARSS Full paper submission is June 1, 2021.
- While submitting a classification result, each team will acknowledge that, should the result be among the winners, at least one team member will participate to the invited session at IGARSS 2021.
Failure to follow any of these rules will automatically make the submission invalid, resulting in the manuscript not being evaluated and disqualification from prize award.
Participants to the Contest are requested not to submit an extended abstract to IGARSS 2021 by the corresponding conference deadline in January 2021. Only contest winners (participants corresponding to the seven best-ranking submissions) will submit a 4-page paper describing their approach to the Contest by April 23, 2021. The received manuscripts will be reviewed by the Award Committee of the Contest, and reviews sent to the winners. Then winners will submit the final version of the 4 full-paper to IGARSS Data Fusion Contest Invited Session by June 1, 2021, for inclusion in the IGARSS Technical Program and Proceedings.
Acknowledgements
The IADF TC chairs would like to thank the IEEE GRSS for continuously supporting the annual Data Fusion Contest through funding and resources. Original Copernicus Sentinel Data 2020 are available from the European Space Agency (sentinel.esa.int/).
The winners of the competition will receive a total of $10k as prizes, courtesy of the organizing partners.
Contest Terms and Conditions
The data are provided for the purpose of participation in the 2021 Data Fusion Contest. Participants acknowledge that they have read and agree to the following Contest Terms and Conditions:
- In any scientific publication using the data, the data shall be referenced as follows: “[REF. NO.] 2021 IEEE GRSS Data Fusion Contest. Online: www.grss-ieee.org/community/technical-committees/data-fusion”.
- Any scientific publication using the data shall include a section “Acknowledgement”. This section shall include the following sentence: “The authors would like to thank the IEEE GRSS Image Analysis and Data Fusion Technical Committee, Hewlett Packard Enterprise, SolarAid, and Data Science Experts for organizing the Data Fusion Contest.
