AI Challenge

February 6, 2025

Dear AI Challenge Participants,

We hope you’re doing well! We wanted to inform you of some important updates regarding the competition:

1. Weekly Office Hours: Starting next week, we will be holding weekly office hours on Zoom every Wednesday from 11:00 AM to 11:30 AM ET. To avoid creating any unfair advantage, we request that discussions about potential approaches to the challenge problem are kept to the forums, but office hours will be a great opportunity to ask clarifying and logistical questions (e.g. about the data or issues with code submission). We encourage you to join if you need assistance!
   
2. Platform Transition to Codabench: As of today, we are transitioning from EvalAI to Codabench for the remainder of the competition. As part of this change, you will now need to submit your code using a conda environment. This new platform will better suit our evaluation needs and will provide a smoother experience moving forward.

With this transition, you will no longer need to update Docker images. Instead, you will manage dependencies using an environment.yml file. We will use this file to install all necessary dependencies for your solution. Specifically, we will create the conda environment using micromamba (due to its high performance) and then execute submission.py, which should remain the entry point/controller of your model.

We strongly encourage you to start adapting your solutions to this new environment and testing them locally to ensure that micromamba can correctly install all dependencies. We will provide a more detailed explanation of the submission process in the coming days as we finish migrating the platform.
We understand that this transition may cause some inconvenience, especially at this stage in the competition, and we apologize for the disruption. Unfortunately, EvalAI was unable to resolve the evaluation issues we encountered, and we believe Codabench will be much more reliable for your submissions.

What’s Next?
We will be adding detailed documentation about this change and the conda environment setup to the STORM-AI wiki later this week. You can now access the challenge on Codabench here. We will send out another email when the wiki is updated with all of the new submission instructions, but in the meantime feel free to submit your code to Codabench for evaluation if you are comfortable doing so.

Additionally, we will do our best to provide strong support to help you adapt to the new requirements and platform as smoothly as possible. We encourage you to participate actively in the forums and ask any questions you may have.

Thank you for your understanding and continued participation. We look forward to seeing your submissions and hope to see you at our weekly office hours!

Best regards,
Your AI Challenge Organizers 

December 17, 2024

Dear challenge participants,

Welcome to Phase 1 of the 2025 MIT ARCLab Prize for AI Innovation in Space! You can now submit your code on a rolling basis for evaluation on the EvalAI platform. You will not be able to see your scores on the leaderboard quite yet, but we should have full functionality on EvalAI within the next few days.

The training dataset, model submission tutorial, development toolkit, and baseline model are available at the following links:

• Dataset: it currently contains initial satellite orbital states between the years 2000 and 2020 along with a 60-day OMNI2 space weather data fragment and 3 days of orbit-averaged thermospheric densities for each initial state. The corresponding 60-day GOES data fragments will be uploaded before the end of the week. We may also add more test cases to increase the size of the training dataset and will keep you posted.
  
• Development toolkit: the Satellite Tracking and Orbit Resilience Modeling with AI (STORM-AI) DevKit accessible on GitHub, reporting tutorials, baseline model, and a high-fidelity orbit propagator.
  
• Baseline model: this script represents the baseline model which appears on the current EvalAI leaderboard.
  
• Model submission tutorial: this section of the wiki discusses Docker containers and example submissions for models coded in Python as well as other helpful information about submitting your model to EvalAI for evaluation.

As a reminder, the STORM-AI Wiki Page reports extensive documentation about the challenge dataset, example code, tutorials, EvalAI submission process, supplemental resources, and FAQs.

If you have any questions about the data or development toolkit, please feel free to reach out via our devkit Q&A. For general inquiries, you can email us at ai_challenge@mit.edu.

We’ll keep you informed via email of any changes to the dataset or EvalAI page over the next few days. Good luck!

Best Regards,
Your AI Challenge Organizers

December 11, 2024

Good evening challenge participants,

We hope that you’re ready for the competition phase because the warm-up period is nearly over! Our baseline model will be available on the devkit this weekend after we make some final adjustments, and you will soon have access to our EvalAI page and the full training dataset. 

In preparation for Phase 1, we have also provided a new version of the warm-up dataset that is organized into I/O pairs. We hope this will help streamline the process of training your algorithms. The new format is consistent with our evaluation pipeline, so the training data that will be released during Phase 1 will follow this convention as well. Changes from the V1 data format include:

• Satellite geodetic position coordinates for each set of initial orbital elements have been added to the initial state file (wu001_to_wu715-initial_states.csv)
  
• GOES & OMNI2 data has been split into 60-day chunks so that each initial state has a corresponding GOES input file and OMNI2 input file with the specified file ID (“wu###” for the warmup dataset)
  
• SWARM density data has been split into 3-day chunks so that each initial state has a corresponding “forecasted” density file for model validation
  
• SWARM density data columns that do not appear in the required output format have been removed, meaning that for each initial state, your model should output a 3-day density forecast in the same format as the files in the Sat_Density folder
  
• The STORM-AI dropbox folder now has the following structure:
  
  📁 STORM-AI
├── 📁 warmup
│   ├── 📁 v1
│   └── 📁 v2
│       ├── 📁 GOES
│       ├── 📁 OMNI2
│       ├── 📁 Sat_Density
│       └── 📝 wu001_to_wu715-initial_states.csv
└── 📁 phase_1

***Please note that the reformatted inputs and outputs still contain gaps due to the nature of the source data***

If you have any questions about the data or development toolkit, please feel free to reach out via our devkit Q&A. For general inquiries, you can email us at ai_challenge@mit.edu.

Best Regards,
Your AI Challenge Organizers

November 15, 2024

Dear challenge participants,

Good evening and welcome to the warm-up phase of the 2025 MIT ARCLab Prize for AI Innovation in Space! Phase 0 launched earlier today, and there are a few key updates and resources that we want to make you aware of.

TL;DR: Competition Overview | DevKit | Dataset | Wiki | Q&A

EvalAI Development Status
As in last year’s competition, we will begin accepting and evaluating your submissions through EvalAI at the launch of Phase 1 in December. We previously announced that the new challenge would be available for you to view on the EvalAI platform during this warm-up period; however, we are still working with the EvalAI team to validate our evaluation pipeline. We’ll give you a heads up when we go live on EvalAI, but in the meantime, we have published a comprehensive challenge overview to the ARCLab website at the link above.

STORM-AI Development Kit
To help you get started on your models, we created the Satellite Tracking and Orbit Resilience Modeling with AI (STORM-AI) DevKit on GitHub. The repo currently includes a transformer-based atmospheric density forecaster based on this paper, which was written by an ARCLab alum and the original inspiration for this year’s challenge problem. In the coming weeks, we will continue to release tutorials and other components of the devkit, including the code for our baseline model that will appear on the Phase 1 leaderboard. We will notify you of new content via email and do our best to respond to any questions or feedback on the devkit Q&A.

Warm-Up Dataset
A portion of the public challenge dataset is currently available for download via Dropbox at the link above. It includes a few years of space weather data and initial satellite states that your models should parse as inputs. There is also a history of orbit-averaged atmospheric density values reported by ESA’s SWARM A satellite, which you may use to validate your model. The challenge data formats should change very little from this preliminary dataset and will be finalized well in advance of the Phase 1 launch date. More detailed information about the data is located in the devkit wiki.

STORM-AI Wiki Page
Extensive documentation about the challenge dataset, example code, EvalAI submission process, and supplemental resources is available on the devkit wiki. The latest long-form tutorials, FAQs, and more will live on the wiki throughout the course of the competition, so refer back to it often!

Thank you for being a part of this year’s AI Challenge! We hope that these resources will provide everything you need to start working with the dataset and developing your prediction models. Feel free to reach out with questions about the data or development toolkit via our devkit Q&A, or for general inquiries, email us at ai_challenge@mit.edu.

Best regards,
Your AI Challenge organizers

November 4, 2024

Hello everyone!

Thank you for your interest in the 2025 MIT ARCLab Prize for AI Innovation in Space! 

We are excited to announce that we are launching Phase 0 of our space weather challenge on Friday November 15th. This is a warm-up phase, during which you will be able to view the competition’s EvalAI page, a preliminary GitHub devkit, and a portion of the dataset you will be provided for training your algorithms. Last year’s webpage, devkit, and warm-up dataset are good references for what you can expect.

Code submissions will not be accepted until the launch of Phase 1 in mid-December, but the warm-up phase will give you the opportunity to familiarize yourself with the data types and formats you’ll be asked to work with during the challenge.

Read about the challenge objective, phases, and prize breakdown in our launch announcement or contact the organizers at ai_challenge@mit.edu for more information.

Best regards,
Your AI Challenge organizers